[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:28 dist-test-slave-lqrk krb5kdc[1917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:28 dist-test-slave-lqrk krb5kdc[1917](info): set up 2 sockets
Sep 26 03:37:28 dist-test-slave-lqrk krb5kdc[1917](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 03:37:29.643597 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:29 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:29.807502 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46549
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:30.120074 1930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:30.120374 1930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:30.120505 1930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:30.127734 1930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:30.127843 1930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:30.127918 1930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:30.127993 1930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:30.128051 1930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:30.137957 1930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:30.139297 1930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:30.140618 1930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:30.152331 1935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:30.152292 1936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:30.152370 1938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:30.152828 1930 server_base.cc:1034] running on GCE node
I20260926 03:37:30.153645 1930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:30.155160 1930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:30.156358 1930 hybrid_clock.cc:648] HybridClock initialized: now 1790393850156264 us; error 88 us; skew 500 ppm
I20260926 03:37:30.156700 1930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:30 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:30.159713 1930 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:30.161240 1930 webserver.cc:460] Webserver started at http://127.1.221.190:42109/ using document root <none> and password file <none>
I20260926 03:37:30.161973 1930 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:30.162088 1930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:30.162367 1930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:30.165398 1930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "9160613702f6440fa30af5ce6dfa1c78"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "91dfb5cf9f2e9a401fc1d095e391c699"
server_key_iv: "32d74c9e363546ffeae9a0bb1e1cd385"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.166154 1930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "9160613702f6440fa30af5ce6dfa1c78"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "91dfb5cf9f2e9a401fc1d095e391c699"
server_key_iv: "32d74c9e363546ffeae9a0bb1e1cd385"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.171623 1930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:37:30.175195 1945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:30.176620 1930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:30.176788 1930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "9160613702f6440fa30af5ce6dfa1c78"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "91dfb5cf9f2e9a401fc1d095e391c699"
server_key_iv: "32d74c9e363546ffeae9a0bb1e1cd385"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.176978 1930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:30.199319 1930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:30.203677 1930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:30.203950 1930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:30.221374 1930 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:46549
I20260926 03:37:30.221426 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:46549 every 8 connection(s)
I20260926 03:37:30.222782 1930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:30.226761 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:30.233958 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 1930
I20260926 03:37:30.234302 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:30.236867 1910 external_mini_cluster.cc:1428] Setting key bbf59fe5b504b06a35ebfabfc9bbecb3
I20260926 03:37:30.239367 1997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: Bootstrap starting.
I20260926 03:37:30.243340 1997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:30.244529 1997 log.cc:824] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:30.247489 1997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: No bootstrap required, opened a new log
I20260926 03:37:30.257640 1997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } }
I20260926 03:37:30.258196 1997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:30.258322 1997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9160613702f6440fa30af5ce6dfa1c78, State: Initialized, Role: FOLLOWER
I20260926 03:37:30.259061 1997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } }
I20260926 03:37:30.259305 1997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:30.259488 1997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:30.259667 1997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:30.262691 1997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } }
I20260926 03:37:30.263265 1997 leader_election.cc:302] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9160613702f6440fa30af5ce6dfa1c78; no voters:
I20260926 03:37:30.263958 1997 leader_election.cc:288] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:30.264108 2002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:30.266057 1997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 03:37:30 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:30.269797 2002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [term 1 LEADER]: Becoming Leader. State: Replica: 9160613702f6440fa30af5ce6dfa1c78, State: Running, Role: LEADER
I20260926 03:37:30.270552 2002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } }
I20260926 03:37:30.274322 2003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9160613702f6440fa30af5ce6dfa1c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } } }
I20260926 03:37:30.274750 2003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:30.275315 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9160613702f6440fa30af5ce6dfa1c78. Latest consensus state: current_term: 1 leader_uuid: "9160613702f6440fa30af5ce6dfa1c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9160613702f6440fa30af5ce6dfa1c78" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46549 } } }
I20260926 03:37:30.275669 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:30.281038 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:30.241733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.1.221.190:46549)
0926 03:37:30.242708 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:30.242725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:30.244006 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 03:37:30.252590 (+ 8584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:30.252621 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:30.253257 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:30.253756 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:30.256677 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.258767 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.260790 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.261982 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.262187 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:30.269971 (+ 7784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:30.269987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:30.270013 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:30.270075 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:30.274706 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:30.276004 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:30.276018 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:30.276027 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:30.276274 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:30.276858 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:30.276864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:30.276867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:30.277494 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:30.277750 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:30.278856 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0926 03:37:30.279363 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":430,"threads_started":1}
W20260926 03:37:30.281571 2016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:37:30.281704 2016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:37:30.281836 2017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:30.283829 2017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:30.292158 2017 catalog_manager.cc:1347] Generated new cluster ID: 62ce55a5b01146b682ab23c7e9b7eb09
I20260926 03:37:30.292285 2017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:30.319916 2017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:30.321252 2017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:30.328940 2017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9160613702f6440fa30af5ce6dfa1c78: Generated new TSK 0
I20260926 03:37:30.329779 2017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:30.376315 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:30.692775 2023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:30.693084 2023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:30.693224 2023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:30.700523 2023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:30.700671 2023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:30.700825 2023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:30.711087 2023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:30.712478 2023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:30.713874 2023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:30.727101 2029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:30.727144 2028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:30.727120 2031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:30.733235 2023 server_base.cc:1034] running on GCE node
I20260926 03:37:30.733978 2023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:30.735021 2023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:30.736243 2023 hybrid_clock.cc:648] HybridClock initialized: now 1790393850736180 us; error 61 us; skew 500 ppm
I20260926 03:37:30.736572 2023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:30 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:30.739821 2023 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:30.741534 2023 webserver.cc:460] Webserver started at http://127.1.221.129:39189/ using document root <none> and password file <none>
I20260926 03:37:30.742300 2023 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:30.742421 2023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:30.742717 2023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:30.745820 2023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "bc7b6e5f00bb4626b496e95487367252"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "df8e1edede673b544529cd144abb77da"
server_key_iv: "d7cc4123dcd77d69581a48b43b45d36c"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.746580 2023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "bc7b6e5f00bb4626b496e95487367252"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "df8e1edede673b544529cd144abb77da"
server_key_iv: "d7cc4123dcd77d69581a48b43b45d36c"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.752125 2023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:37:30.755795 2038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:30.757231 2023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:37:30.757395 2023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "bc7b6e5f00bb4626b496e95487367252"
format_stamp: "Formatted at 2026-09-26 03:37:30 on dist-test-slave-lqrk"
server_key: "df8e1edede673b544529cd144abb77da"
server_key_iv: "d7cc4123dcd77d69581a48b43b45d36c"
server_key_version: "encryptionkey@0"
I20260926 03:37:30.757602 2023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:30.787241 2023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:30.791308 2023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:30.791611 2023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:30.792589 2023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:30.793955 2023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:30.794034 2023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:30.794149 2023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:30.794199 2023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:30.823659 2023 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43637
I20260926 03:37:30.823695 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43637 every 8 connection(s)
I20260926 03:37:30.824903 2023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:30.834910 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2023
I20260926 03:37:30.835129 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:30.835824 1910 external_mini_cluster.cc:1428] Setting key f5a434f4f44d117e6f03e73e60915df0
Sep 26 03:37:30 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:30.845113 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:30.827488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47325 (local address 127.1.221.190:46549)
0926 03:37:30.827919 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:30.827924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:30.828923 (+ 999us) server_negotiation.cc:407] Connection header received
0926 03:37:30.830764 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:30.830769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:30.830848 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:30.830943 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:30.832463 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.833152 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.834674 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.835102 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.835209 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:30.839043 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:30.839049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:30.839052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:30.839084 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:30.841725 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:30.842582 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:30.842587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:30.842589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:30.842720 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:30.843178 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:30.843182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:30.843184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:30.843516 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:30.843657 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:30.844434 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 03:37:30.844694 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260926 03:37:30.846087 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:30.828085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:46549 (local address 127.1.221.129:47325)
0926 03:37:30.828770 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:30.828808 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:30.830503 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:30.831191 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:30.831199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:30.831753 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:30.832303 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:30.832322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.833299 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.833302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:30.834548 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:30.834560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:30.835306 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:30.835312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:30.835563 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:30.836194 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:30.836228 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:30.838803 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:30.841901 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:30.841912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:30.841931 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:30.842443 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:30.842833 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:30.842839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:30.842843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:30.843047 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:30.843656 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:30.843666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:30.844029 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 03:37:30.844291 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 03:37:30.844842 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":291,"threads_started":1}
I20260926 03:37:30.847767 2151 heartbeater.cc:347] Connected to a master server at 127.1.221.190:46549
I20260926 03:37:30.848145 2151 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:30.849073 2151 heartbeater.cc:511] Master 127.1.221.190:46549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:37:30.851438 1962 ts_manager.cc:194] Registered new tserver with Master: bc7b6e5f00bb4626b496e95487367252 (127.1.221.129:43637)
I20260926 03:37:30.852955 1962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:47325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:30.874853 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:31.189507 2157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:31.189800 2157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:31.189924 2157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:31.197148 2157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:31.197265 2157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:31.197414 2157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:31.207234 2157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:31.208600 2157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:31.209966 2157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:31.223230 2163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:31.223652 2165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:31.223770 2162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:31.224174 2157 server_base.cc:1034] running on GCE node
I20260926 03:37:31.224781 2157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:31.225745 2157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:31.227007 2157 hybrid_clock.cc:648] HybridClock initialized: now 1790393851226938 us; error 61 us; skew 500 ppm
I20260926 03:37:31.227315 2157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:31.230537 2157 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:31.232089 2157 webserver.cc:460] Webserver started at http://127.1.221.130:45299/ using document root <none> and password file <none>
I20260926 03:37:31.232753 2157 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:31.232864 2157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:31.233155 2157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:31.236266 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "c0c3d3943f07469ebcde12dc4e107bbf"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "866249f0f509ff30fc232dc52af71ea7"
server_key_iv: "2db8dab3f738a435524220183d32786d"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.237061 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "c0c3d3943f07469ebcde12dc4e107bbf"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "866249f0f509ff30fc232dc52af71ea7"
server_key_iv: "2db8dab3f738a435524220183d32786d"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.243903 2157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:37:31.248504 2172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:31.249936 2157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:31.250097 2157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "c0c3d3943f07469ebcde12dc4e107bbf"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "866249f0f509ff30fc232dc52af71ea7"
server_key_iv: "2db8dab3f738a435524220183d32786d"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.250300 2157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:31.268476 2157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:31.272598 2157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:31.272866 2157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:31.273816 2157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:31.275182 2157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:31.275254 2157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:31.275369 2157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:31.275461 2157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:37:31.304960 2157 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45057
I20260926 03:37:31.304993 2284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45057 every 8 connection(s)
I20260926 03:37:31.306255 2157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:31.312047 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2157
I20260926 03:37:31.312252 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:31.312665 1910 external_mini_cluster.cc:1428] Setting key ac4863dadf23d51ad60907ef00dd348d
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:31.326714 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.308802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:58881 (local address 127.1.221.190:46549)
0926 03:37:31.308983 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:31.308988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:31.310178 (+ 1190us) server_negotiation.cc:407] Connection header received
0926 03:37:31.312211 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:31.312217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:31.312304 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:31.312420 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:31.314294 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.315281 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.316875 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.317419 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.317549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.320995 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:31.321001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:31.321004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:31.321038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:31.323534 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.324389 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.324393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.324395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.324518 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.324968 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.324971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.324973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.325223 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:31.325329 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:31.326242 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 03:37:31.326434 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:37:31.328290 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.309270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:46549 (local address 127.1.221.130:58881)
0926 03:37:31.310008 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:31.310054 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:31.311834 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:31.312699 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:31.312708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:31.313337 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:31.314113 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:31.314140 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.315457 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.315462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:31.316728 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:31.316741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.317622 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.317627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:31.317827 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.318447 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:31.318474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:31.320768 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:31.323665 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:31.323674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:31.323693 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:31.324273 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:31.324624 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:31.324629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:31.324633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:31.324873 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:31.325338 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:31.325347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:31.325832 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 03:37:31.326178 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 03:37:31.326771 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":275,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:37:31.329921 2285 heartbeater.cc:347] Connected to a master server at 127.1.221.190:46549
I20260926 03:37:31.330250 2285 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:31.331058 2285 heartbeater.cc:511] Master 127.1.221.190:46549 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:37:31.332871 1962 ts_manager.cc:194] Registered new tserver with Master: c0c3d3943f07469ebcde12dc4e107bbf (127.1.221.130:45057)
I20260926 03:37:31.333709 1962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:58881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:31.351095 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:31.660961 2290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:31.661260 2290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:31.661399 2290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:31.668689 2290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:31.668817 2290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:31.668974 2290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:31.678682 2290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:46549
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:31.680023 2290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:31.681371 2290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:31.694028 2295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:31.693959 2298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:31.694032 2296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:31.694761 2290 server_base.cc:1034] running on GCE node
I20260926 03:37:31.695475 2290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:31.696492 2290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:31.697731 2290 hybrid_clock.cc:648] HybridClock initialized: now 1790393851697671 us; error 48 us; skew 500 ppm
I20260926 03:37:31.698091 2290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:31.701185 2290 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:31.702590 2290 webserver.cc:460] Webserver started at http://127.1.221.131:33623/ using document root <none> and password file <none>
I20260926 03:37:31.703258 2290 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:31.703369 2290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:31.703691 2290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:31.706548 2290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "9e2570e2cf9240acbd9f2a9c0deb6f80"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "30329ab52a1428df2ad86984e75cb999"
server_key_iv: "35d5fdb32f5820d451bdda16ea911885"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.707293 2290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "9e2570e2cf9240acbd9f2a9c0deb6f80"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "30329ab52a1428df2ad86984e75cb999"
server_key_iv: "35d5fdb32f5820d451bdda16ea911885"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.712703 2290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:37:31.716369 2305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:31.717830 2290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:31.718005 2290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "9e2570e2cf9240acbd9f2a9c0deb6f80"
format_stamp: "Formatted at 2026-09-26 03:37:31 on dist-test-slave-lqrk"
server_key: "30329ab52a1428df2ad86984e75cb999"
server_key_iv: "35d5fdb32f5820d451bdda16ea911885"
server_key_version: "encryptionkey@0"
I20260926 03:37:31.718210 2290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:31.735313 2290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:31.739100 2290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:31.739355 2290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:31.740316 2290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:31.741617 2290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:31.741693 2290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:31.741803 2290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:31.741853 2290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:31.772204 2290 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36845
I20260926 03:37:31.772249 2417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36845 every 8 connection(s)
I20260926 03:37:31.773459 2290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:31.776326 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2290
I20260926 03:37:31.776512 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:31.776955 1910 external_mini_cluster.cc:1428] Setting key 1a18b09f003e02f500f243aecd7693b3
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:31.793751 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.776053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:47427 (local address 127.1.221.190:46549)
0926 03:37:31.776263 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:31.776270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:31.777568 (+ 1298us) server_negotiation.cc:407] Connection header received
0926 03:37:31.779667 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:31.779673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:31.779784 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:31.779893 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:31.781490 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.782211 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.783750 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.784196 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.784297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.787825 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:31.787830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:31.787834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:31.787871 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:31.790735 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.791484 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.791489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.791491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.791619 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.792122 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.792127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.792130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.792464 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:31.792579 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:31.793342 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 03:37:31.793511 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:37:31.794742 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.776589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:46549 (local address 127.1.221.131:47427)
0926 03:37:31.777382 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:31.777435 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:31.779374 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:31.780079 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:31.780091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:31.780668 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:31.781349 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:31.781370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.782326 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.782330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:31.783616 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:31.783628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.784424 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.784430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:31.784648 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.785303 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:31.785328 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:31.787614 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:31.790881 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:31.790889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:31.790903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:31.791344 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:31.791730 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:31.791734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:31.791736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:31.792013 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:31.792587 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:31.792595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:31.792986 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 03:37:31.793240 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:37:31.793645 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260926 03:37:31.796062 2418 heartbeater.cc:347] Connected to a master server at 127.1.221.190:46549
I20260926 03:37:31.796315 2418 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:31.796973 2418 heartbeater.cc:511] Master 127.1.221.190:46549 requested a full tablet report, sending...
I20260926 03:37:31.798225 1962 ts_manager.cc:194] Registered new tserver with Master: 9e2570e2cf9240acbd9f2a9c0deb6f80 (127.1.221.131:36845)
I20260926 03:37:31.799000 1962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:47427
I20260926 03:37:31.805186 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:31.849181 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.833646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.1.221.129:43637)
0926 03:37:31.834335 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:31.834343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:31.834433 (+ 90us) server_negotiation.cc:407] Connection header received
0926 03:37:31.834621 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:31.834628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:31.834893 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:31.835034 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:31.836953 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.837998 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.839296 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.839918 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.840029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.843780 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:31.843792 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:31.843808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:31.843843 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:31.846583 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.847315 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.847322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.847325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.847509 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.847834 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.847837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.847839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.848179 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:31.848409 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:31.848622 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:37:31.848821 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":328,"threads_started":1}
W20260926 03:37:31.850447 2066 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52912
I20260926 03:37:31.855901 2151 heartbeater.cc:503] Master 127.1.221.190:46549 was elected leader, sending a full tablet report...
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:37:31 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:31.892004 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.878158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.1.221.129:43637)
0926 03:37:31.878400 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:31.878407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:31.878524 (+ 117us) server_negotiation.cc:407] Connection header received
0926 03:37:31.878722 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:31.878726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:31.878821 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:31.878967 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:31.880790 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.881521 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.882709 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.883143 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.883255 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.887044 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:31.887049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:31.887053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:31.887088 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:31.889698 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.890313 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.890318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.890321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.890480 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.890785 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.890788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.890789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.891074 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:31.891282 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:31.891574 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:37:31.891750 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:37:31.893628 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2023
I20260926 03:37:31.906689 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:31.894564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.1.221.129:43637)
0926 03:37:31.894728 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:31.894732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:31.895019 (+ 287us) server_negotiation.cc:407] Connection header received
0926 03:37:31.895126 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:31.895129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:31.895185 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:31.895264 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:31.896369 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.897069 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.898323 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:31.898718 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:31.898802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:31.901074 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:31.901080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:31.901083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:31.901115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:31.904281 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.905001 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.905005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.905007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.905139 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:31.905412 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:31.905415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:31.905417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:31.905662 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:31.905833 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:31.906158 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 03:37:31.906323 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:37:31.907455 2146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:31.990439 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2023
I20260926 03:37:32.004491 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2157
Sep 26 03:37:32 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:37:32.020735 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:32.005313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.1.221.130:45057)
0926 03:37:32.005881 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:32.005893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:32.005929 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:37:32.006037 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:32.006044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:32.006271 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:32.006406 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:32.007574 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:32.008607 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:32.010066 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:32.010632 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:32.010738 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:32.014845 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:32.014891 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:32.014908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:32.014953 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:32.017765 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:32.018471 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:32.018485 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:32.018488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:32.018635 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:32.019042 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:32.019048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:32.019055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:32.019551 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:32.019786 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:32.020127 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:37:32.020376 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":312,"threads_started":1}
I20260926 03:37:32.022097 2280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:32.105253 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2157
I20260926 03:37:32.122473 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2290
Sep 26 03:37:32 dist-test-slave-lqrk krb5kdc[1917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:37:32.138335 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:32.123555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.1.221.131:36845)
0926 03:37:32.124169 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:32.124176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:32.124205 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:37:32.124268 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:32.124274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:32.124489 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:32.124630 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:32.125822 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:32.126842 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:32.128290 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:32.128861 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:32.128961 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:32.132829 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:32.132843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:32.132863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:32.132910 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:32.135702 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:32.136384 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:32.136392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:32.136395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:32.136527 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:32.136882 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:32.136886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:32.136888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:32.137222 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:32.137454 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:32.137813 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:37:32.138033 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
I20260926 03:37:32.139631 2413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:32.220240 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2290
I20260926 03:37:32.233415 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 1930
I20260926 03:37:32.234611 1992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:32.335206 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 1930
2026-09-26T03:37:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3703 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:32 dist-test-slave-lqrk krb5kdc[2439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:32 dist-test-slave-lqrk krb5kdc[2439](info): set up 2 sockets
Sep 26 03:37:32 dist-test-slave-lqrk krb5kdc[2439](info): commencing operation
krb5kdc: starting...
W20260926 03:37:33.319578 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:33 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:33.493217 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41381
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:33.798770 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:33.799064 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:33.799173 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:33.806399 2452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:33.806519 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:33.806572 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:33.806649 2452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:33.806707 2452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:33.816610 2452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:33.817963 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:33.819325 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:33.830840 2460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:33.830914 2457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:33.830943 2458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:33.831714 2452 server_base.cc:1034] running on GCE node
I20260926 03:37:33.832557 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:33.834077 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:33.835295 2452 hybrid_clock.cc:648] HybridClock initialized: now 1790393853835246 us; error 42 us; skew 500 ppm
I20260926 03:37:33.835666 2452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:33 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:33.844122 2452 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:33.845667 2452 webserver.cc:460] Webserver started at http://127.1.221.190:42621/ using document root <none> and password file <none>
I20260926 03:37:33.846357 2452 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:33.846460 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:33.846725 2452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:33.849854 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "2f78b3d7a69d4b23a0317c297842c475"
format_stamp: "Formatted at 2026-09-26 03:37:33 on dist-test-slave-lqrk"
server_key: "a5def0df06997124f1db4fbcf373f0e0"
server_key_iv: "6e273b2af391c61dfd95e76293a390bd"
server_key_version: "encryptionkey@0"
I20260926 03:37:33.850569 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "2f78b3d7a69d4b23a0317c297842c475"
format_stamp: "Formatted at 2026-09-26 03:37:33 on dist-test-slave-lqrk"
server_key: "a5def0df06997124f1db4fbcf373f0e0"
server_key_iv: "6e273b2af391c61dfd95e76293a390bd"
server_key_version: "encryptionkey@0"
I20260926 03:37:33.855834 2452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:37:33.859326 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:33.860772 2452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:33.860940 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "2f78b3d7a69d4b23a0317c297842c475"
format_stamp: "Formatted at 2026-09-26 03:37:33 on dist-test-slave-lqrk"
server_key: "a5def0df06997124f1db4fbcf373f0e0"
server_key_iv: "6e273b2af391c61dfd95e76293a390bd"
server_key_version: "encryptionkey@0"
I20260926 03:37:33.861135 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:33.876298 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:33.880218 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:33.880475 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:33.897333 2452 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41381
I20260926 03:37:33.897356 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41381 every 8 connection(s)
I20260926 03:37:33.898694 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:33.902354 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:33.908182 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2452
I20260926 03:37:33.908419 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:33.909199 1910 external_mini_cluster.cc:1428] Setting key 8ff4daf52cb35b0edbf16596d959daca
I20260926 03:37:33.913429 2519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475: Bootstrap starting.
I20260926 03:37:33.917001 2519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:33.918084 2519 log.cc:824] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:33.921128 2519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475: No bootstrap required, opened a new log
Sep 26 03:37:33 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:33.929015 2519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } }
I20260926 03:37:33.929399 2519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:33.929498 2519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f78b3d7a69d4b23a0317c297842c475, State: Initialized, Role: FOLLOWER
I20260926 03:37:33.930105 2519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } }
I20260926 03:37:33.930368 2519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:33.930496 2519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:33.930665 2519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:33.931602 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:33.911277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.1.221.190:41381)
0926 03:37:33.912068 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:33.912081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:33.912123 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:37:33.913801 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:33.913824 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:33.914231 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:33.914564 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:33.916253 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:33.917608 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:33.919145 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:33.919871 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:33.920002 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:33.924327 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:33.924336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:33.924355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:33.924395 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:33.927449 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:33.928112 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:33.928119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:33.928126 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:33.928292 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:33.928647 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:33.928652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:33.928654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:33.929187 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:33.929453 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:33.929982 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 03:37:33.930441 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":355,"threads_started":1}
I20260926 03:37:33.933063 2519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } }
I20260926 03:37:33.933538 2519 leader_election.cc:302] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f78b3d7a69d4b23a0317c297842c475; no voters:
I20260926 03:37:33.934099 2519 leader_election.cc:288] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:33.934211 2524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:33.934930 2524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [term 1 LEADER]: Becoming Leader. State: Replica: 2f78b3d7a69d4b23a0317c297842c475, State: Running, Role: LEADER
I20260926 03:37:33.935529 2524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } }
I20260926 03:37:33.936419 2519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:33.938192 2525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f78b3d7a69d4b23a0317c297842c475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } } }
I20260926 03:37:33.938567 2525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:33.940243 2526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f78b3d7a69d4b23a0317c297842c475. Latest consensus state: current_term: 1 leader_uuid: "2f78b3d7a69d4b23a0317c297842c475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f78b3d7a69d4b23a0317c297842c475" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41381 } } }
I20260926 03:37:33.940527 2526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:33.945806 2531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:33.951539 2531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:33.960049 2531 catalog_manager.cc:1347] Generated new cluster ID: 6526a81ec350436ba563192019d518a1
I20260926 03:37:33.960171 2531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:33.973812 2531 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:33.975252 2531 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:33.984308 2531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f78b3d7a69d4b23a0317c297842c475: Generated new TSK 0
I20260926 03:37:33.985102 2531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:34.027205 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:34.340354 2545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:34.340709 2545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:34.340821 2545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:34.347955 2545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:34.348052 2545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:34.348169 2545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:34.357635 2545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:34.358842 2545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:34.360154 2545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:34.373535 2550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:34.373585 2551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:34.373616 2553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:34.374230 2545 server_base.cc:1034] running on GCE node
I20260926 03:37:34.374814 2545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:34.375831 2545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:34.377028 2545 hybrid_clock.cc:648] HybridClock initialized: now 1790393854376980 us; error 42 us; skew 500 ppm
I20260926 03:37:34.377345 2545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:34 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:34.380463 2545 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:34.381829 2545 webserver.cc:460] Webserver started at http://127.1.221.129:42305/ using document root <none> and password file <none>
I20260926 03:37:34.382480 2545 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:34.382573 2545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:34.382815 2545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:34.385636 2545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "8fbad9813e3c452c986d66e41916f537"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "b447c5f02eff2266950ea612f5ab3e97"
server_key_iv: "e32a1b12fb10448bb063287849e6eea2"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.386355 2545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "8fbad9813e3c452c986d66e41916f537"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "b447c5f02eff2266950ea612f5ab3e97"
server_key_iv: "e32a1b12fb10448bb063287849e6eea2"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.391201 2545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:37:34.394459 2560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.395819 2545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:34.395972 2545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "8fbad9813e3c452c986d66e41916f537"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "b447c5f02eff2266950ea612f5ab3e97"
server_key_iv: "e32a1b12fb10448bb063287849e6eea2"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.396154 2545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:34.413978 2545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:34.417974 2545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:34.418236 2545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:34.419204 2545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:34.420615 2545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:34.420691 2545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.420809 2545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:34.420861 2545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.453176 2545 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36921
I20260926 03:37:34.453217 2672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36921 every 8 connection(s)
I20260926 03:37:34.454440 2545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:34.462891 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2545
I20260926 03:37:34.463088 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:34.463783 1910 external_mini_cluster.cc:1428] Setting key 9e6defda04d5084cbf248c38df8114bd
Sep 26 03:37:34 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:34.475651 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:34.457052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60349 (local address 127.1.221.190:41381)
0926 03:37:34.457454 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:34.457459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:34.458371 (+ 912us) server_negotiation.cc:407] Connection header received
0926 03:37:34.460283 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:34.460289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:34.460373 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:34.460483 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:34.462082 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.462805 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.464784 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.465393 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.465533 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:34.469647 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:34.469653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:34.469656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:34.469691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:34.472673 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:34.473374 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:34.473380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:34.473384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:34.473537 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:34.473914 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:34.473919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:34.473921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:34.474267 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:34.474401 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:34.475053 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 03:37:34.475276 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
I20260926 03:37:34.476475 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:34.457508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41381 (local address 127.1.221.129:60349)
0926 03:37:34.458199 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:34.458239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:34.459988 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:34.460749 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:34.460758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:34.461290 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:34.461890 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:34.461910 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.462980 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.462986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:34.464646 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:34.464659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.465633 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.465641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:34.465909 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:34.466631 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:34.466666 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:34.469399 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:34.472807 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:34.472815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:34.472829 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:34.473251 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:34.473632 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:34.473635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:34.473638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:34.473812 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:34.474375 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:34.474381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:34.474704 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 03:37:34.474926 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 03:37:34.475326 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":255,"threads_started":1}
I20260926 03:37:34.477957 2673 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41381
I20260926 03:37:34.478247 2673 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:34.478950 2673 heartbeater.cc:511] Master 127.1.221.190:41381 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:37:34.481036 2484 ts_manager.cc:194] Registered new tserver with Master: 8fbad9813e3c452c986d66e41916f537 (127.1.221.129:36921)
I20260926 03:37:34.482508 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:60349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:34.503465 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:34.812464 2679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:34.812780 2679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:34.812907 2679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:34.820190 2679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:34.820324 2679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:34.820484 2679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:34.830338 2679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:34.831715 2679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:34.833071 2679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:34.846108 2685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:34.846129 2684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:34.846065 2687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:34.847014 2679 server_base.cc:1034] running on GCE node
I20260926 03:37:34.847718 2679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:34.848688 2679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:34.849922 2679 hybrid_clock.cc:648] HybridClock initialized: now 1790393854849852 us; error 56 us; skew 500 ppm
I20260926 03:37:34.850271 2679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:34 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:34.853427 2679 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:34.854846 2679 webserver.cc:460] Webserver started at http://127.1.221.130:36101/ using document root <none> and password file <none>
I20260926 03:37:34.855597 2679 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:34.855715 2679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:34.856002 2679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:34.859023 2679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "753b64deae7e40f8ad4118c396ee192b"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "2a948c43afbe5e14a62c0a33a3c315db"
server_key_iv: "d80ac9e9525922b224fc5cf73ff68c39"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.859840 2679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "753b64deae7e40f8ad4118c396ee192b"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "2a948c43afbe5e14a62c0a33a3c315db"
server_key_iv: "d80ac9e9525922b224fc5cf73ff68c39"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.865413 2679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:34.869011 2694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.870335 2679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:37:34.870500 2679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "753b64deae7e40f8ad4118c396ee192b"
format_stamp: "Formatted at 2026-09-26 03:37:34 on dist-test-slave-lqrk"
server_key: "2a948c43afbe5e14a62c0a33a3c315db"
server_key_iv: "d80ac9e9525922b224fc5cf73ff68c39"
server_key_version: "encryptionkey@0"
I20260926 03:37:34.870690 2679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:34.886188 2679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:34.890501 2679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:34.890763 2679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:34.891767 2679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:34.893045 2679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:34.893123 2679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.893252 2679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:34.893304 2679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:34.923491 2679 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:32911
I20260926 03:37:34.923532 2806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:32911 every 8 connection(s)
I20260926 03:37:34.924726 2679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:34.928514 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2679
I20260926 03:37:34.928709 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:34.929186 1910 external_mini_cluster.cc:1428] Setting key 00bea6698594743e8c06201989e93ff1
Sep 26 03:37:34 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:37:34.945044 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:34.927206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34637 (local address 127.1.221.190:41381)
0926 03:37:34.927400 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:34.927407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:34.928598 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 03:37:34.930532 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:34.930537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:34.930617 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:34.930704 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:34.932308 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.933033 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.934577 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.934984 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.935084 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:34.939194 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:34.939200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:34.939204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:34.939238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:34.941760 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:34.942680 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:34.942684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:34.942686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:34.942815 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:34.943295 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:34.943298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:34.943300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:34.943593 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:34.943706 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:34.944589 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 03:37:34.944790 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:37:34.946241 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:34.927723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41381 (local address 127.1.221.130:34637)
0926 03:37:34.928431 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:34.928470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:34.930258 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:34.930969 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:34.930979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:34.931527 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:34.932137 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:34.932157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.933190 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.933194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:34.934432 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:34.934444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:34.935201 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:34.935209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:34.935548 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:34.936226 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:34.936282 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:34.938947 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:34.941925 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:34.941935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:34.941956 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:34.942556 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:34.942925 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:34.942930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:34.942934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:34.943191 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:34.943713 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:34.943721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:34.944208 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 03:37:34.944520 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 03:37:34.945075 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:37:34.947765 2807 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41381
I20260926 03:37:34.948123 2807 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:34.949029 2807 heartbeater.cc:511] Master 127.1.221.190:41381 requested a full tablet report, sending...
I20260926 03:37:34.950686 2484 ts_manager.cc:194] Registered new tserver with Master: 753b64deae7e40f8ad4118c396ee192b (127.1.221.130:32911)
I20260926 03:37:34.951565 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:34637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:34.970435 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:35.288765 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:35.289068 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:35.289212 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:35.296568 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:35.296684 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:35.296839 2812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:35.306855 2812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:35.308204 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:35.309566 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:35.322461 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:35.322417 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:35.322376 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:35.323478 2812 server_base.cc:1034] running on GCE node
I20260926 03:37:35.324086 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:35.325050 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:35.326249 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790393855326195 us; error 49 us; skew 500 ppm
I20260926 03:37:35.326567 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:35.329607 2812 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:35.331089 2812 webserver.cc:460] Webserver started at http://127.1.221.131:43413/ using document root <none> and password file <none>
I20260926 03:37:35.331826 2812 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:35.331940 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:35.332233 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:35.335210 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "3668ae5798f44fd082485506d3d627a0"
format_stamp: "Formatted at 2026-09-26 03:37:35 on dist-test-slave-lqrk"
server_key: "38070080bb6dd23dddb8d83c34cd602d"
server_key_iv: "5c5ae6eac663319332816373632d74ce"
server_key_version: "encryptionkey@0"
I20260926 03:37:35.336026 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "3668ae5798f44fd082485506d3d627a0"
format_stamp: "Formatted at 2026-09-26 03:37:35 on dist-test-slave-lqrk"
server_key: "38070080bb6dd23dddb8d83c34cd602d"
server_key_iv: "5c5ae6eac663319332816373632d74ce"
server_key_version: "encryptionkey@0"
I20260926 03:37:35.341406 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:35.344926 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:35.346354 2812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:37:35.346535 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "3668ae5798f44fd082485506d3d627a0"
format_stamp: "Formatted at 2026-09-26 03:37:35 on dist-test-slave-lqrk"
server_key: "38070080bb6dd23dddb8d83c34cd602d"
server_key_iv: "5c5ae6eac663319332816373632d74ce"
server_key_version: "encryptionkey@0"
I20260926 03:37:35.346720 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:35.362788 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:35.367175 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:35.367487 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:35.368497 2812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:35.369843 2812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:35.369925 2812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:35.370023 2812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:35.370065 2812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:35.400139 2812 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:39515
I20260926 03:37:35.400187 2939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:39515 every 8 connection(s)
I20260926 03:37:35.401474 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:35.406602 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 2812
I20260926 03:37:35.406774 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:35.407202 1910 external_mini_cluster.cc:1428] Setting key 122d2aaa9147f817f792f2161ee74a07
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:35.420867 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.403952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46321 (local address 127.1.221.190:41381)
0926 03:37:35.404128 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.404132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.405299 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 03:37:35.407159 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.407164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.407250 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.407347 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.409169 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.409952 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.411522 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.411929 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.412038 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.415481 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.415487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.415490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.415524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.417917 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.418716 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.418721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.418724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.418850 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.419244 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.419247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.419248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.419574 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.419690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.420355 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.420565 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:37:35.422232 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.404441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41381 (local address 127.1.221.131:46321)
0926 03:37:35.405147 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.405185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.406876 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.407662 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.407674 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.408218 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:35.408995 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.409017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.410110 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.410114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.411344 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.411357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.412145 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.412151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.412408 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.413032 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:35.413056 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:35.415217 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:35.418063 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:35.418071 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:35.418086 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:35.418591 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:35.418959 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:35.418962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:35.418964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:35.419162 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:35.419673 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:35.419679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:35.420031 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.420296 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.420848 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
I20260926 03:37:35.423607 2940 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41381
I20260926 03:37:35.423914 2940 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:35.424649 2940 heartbeater.cc:511] Master 127.1.221.190:41381 requested a full tablet report, sending...
I20260926 03:37:35.426048 2484 ts_manager.cc:194] Registered new tserver with Master: 3668ae5798f44fd082485506d3d627a0 (127.1.221.131:39515)
I20260926 03:37:35.426895 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:46321
I20260926 03:37:35.435146 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:37:35.453842 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.441557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.1.221.190:41381)
0926 03:37:35.441721 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.441726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.442154 (+ 428us) server_negotiation.cc:407] Connection header received
0926 03:37:35.442296 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.442300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.442362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.442447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.443857 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.444588 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.445866 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.446304 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.446399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.448653 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.448659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.448662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.448696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.451287 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.451969 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.451973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.451976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.452102 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.452395 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.452398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.452400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.452720 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.452962 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.453325 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.453549 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:37:35.460446 2484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:35.467968 2484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:35.485656 2673 heartbeater.cc:503] Master 127.1.221.190:41381 was elected leader, sending a full tablet report...
I20260926 03:37:35.507998 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.480751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.1.221.130:32911)
0926 03:37:35.496487 (+ 15736us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.496497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.497336 (+ 839us) server_negotiation.cc:407] Connection header received
0926 03:37:35.497783 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.497793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.498098 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.498341 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:35.500637 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.502229 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.504967 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.506406 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.506536 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.506738 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.507486 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.507660 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15513,"thread_start_us":15347,"threads_started":1}
I20260926 03:37:35.508028 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.480561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:32911 (local address 127.0.0.1:44424)
0926 03:37:35.497148 (+ 16587us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.497207 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.497480 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.499950 (+ 2470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.499960 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.499999 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:35.500396 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.500409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.502389 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.502394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.504807 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.504817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.506921 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.506927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.507132 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.507190 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.507316 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.507552 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16233,"mutex_wait_us":9,"thread_start_us":17164,"threads_started":1}
I20260926 03:37:35.510587 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.481069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:39515 (local address 127.0.0.1:33688)
0926 03:37:35.502678 (+ 21609us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.502699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.502909 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.503722 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.503726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.503750 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:35.504188 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.504200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.505865 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.505870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.507824 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.507840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.509702 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.509708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.509888 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.509921 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.510012 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.510152 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20830,"mutex_wait_us":23,"thread_start_us":20364,"threads_started":1}
I20260926 03:37:35.511314 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.481413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.1.221.131:39515)
0926 03:37:35.500689 (+ 19276us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.500700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.502803 (+ 2103us) server_negotiation.cc:407] Connection header received
0926 03:37:35.503114 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.503123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.503410 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.503588 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:35.504369 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.505693 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.508011 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.509495 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.509631 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.509876 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.510770 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.510984 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19096,"thread_start_us":18892,"threads_started":1}
I20260926 03:37:35.511572 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.481422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.1.221.129:36921)
0926 03:37:35.501260 (+ 19838us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.501269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.501983 (+ 714us) server_negotiation.cc:407] Connection header received
0926 03:37:35.502109 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.502119 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.502513 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.502707 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:35.503785 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.505120 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.507409 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.510025 (+ 2616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.510157 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.510384 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.511049 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.511228 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19659,"thread_start_us":19461,"threads_started":1}
I20260926 03:37:35.512336 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.480805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36921 (local address 127.0.0.1:49006)
0926 03:37:35.501686 (+ 20881us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.501705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.501836 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.502987 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.502991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.503014 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:35.503562 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.503575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.505249 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.505254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.507220 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.507231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.510553 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.510561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.510860 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.510892 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.510978 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.511115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20755,"mutex_wait_us":57,"thread_start_us":20961,"threads_started":1}
I20260926 03:37:35.512773 2742 tablet_service.cc:1467] Processing CreateTablet for tablet f2a670cf67d04baeb40f5b27b6f1e6c8 (DEFAULT_TABLE table=test-table [id=5591e1474806458fa980d80b64d2d2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:35.514365 2742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a670cf67d04baeb40f5b27b6f1e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:35.515185 2875 tablet_service.cc:1467] Processing CreateTablet for tablet f2a670cf67d04baeb40f5b27b6f1e6c8 (DEFAULT_TABLE table=test-table [id=5591e1474806458fa980d80b64d2d2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:35.515365 2608 tablet_service.cc:1467] Processing CreateTablet for tablet f2a670cf67d04baeb40f5b27b6f1e6c8 (DEFAULT_TABLE table=test-table [id=5591e1474806458fa980d80b64d2d2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:35.516728 2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a670cf67d04baeb40f5b27b6f1e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:35.517369 2608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a670cf67d04baeb40f5b27b6f1e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:35.529291 2958 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Bootstrap starting.
I20260926 03:37:35.531283 2959 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Bootstrap starting.
I20260926 03:37:35.532986 2960 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Bootstrap starting.
I20260926 03:37:35.534086 2958 tablet_bootstrap.cc:654] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:35.534850 2959 tablet_bootstrap.cc:654] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:35.536079 2959 log.cc:824] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:35.536079 2958 log.cc:824] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:35.536152 2960 tablet_bootstrap.cc:654] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:35.537317 2960 log.cc:824] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:35.539585 2958 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: No bootstrap required, opened a new log
I20260926 03:37:35.539584 2959 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: No bootstrap required, opened a new log
I20260926 03:37:35.539865 2959 ts_tablet_manager.cc:1397] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 03:37:35.539861 2958 ts_tablet_manager.cc:1397] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:37:35.541627 2960 tablet_bootstrap.cc:492] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: No bootstrap required, opened a new log
I20260926 03:37:35.541874 2960 ts_tablet_manager.cc:1397] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 03:37:35.549425 2960 raft_consensus.cc:348] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.549830 2960 raft_consensus.cc:374] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:35.549952 2960 raft_consensus.cc:729] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3668ae5798f44fd082485506d3d627a0, State: Initialized, Role: FOLLOWER
I20260926 03:37:35.550376 2959 raft_consensus.cc:348] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.550580 2960 consensus_queue.cc:260] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.550843 2959 raft_consensus.cc:374] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:35.550998 2959 raft_consensus.cc:729] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753b64deae7e40f8ad4118c396ee192b, State: Initialized, Role: FOLLOWER
I20260926 03:37:35.550812 2958 raft_consensus.cc:348] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.551273 2958 raft_consensus.cc:374] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:35.551568 2958 raft_consensus.cc:729] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbad9813e3c452c986d66e41916f537, State: Initialized, Role: FOLLOWER
I20260926 03:37:35.551800 2959 consensus_queue.cc:260] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.552424 2940 heartbeater.cc:503] Master 127.1.221.190:41381 was elected leader, sending a full tablet report...
I20260926 03:37:35.552330 2958 consensus_queue.cc:260] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.552570 2960 ts_tablet_manager.cc:1428] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260926 03:37:35.554090 2958 ts_tablet_manager.cc:1428] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:37:35.556421 2807 heartbeater.cc:503] Master 127.1.221.190:41381 was elected leader, sending a full tablet report...
I20260926 03:37:35.557749 2959 ts_tablet_manager.cc:1428] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260926 03:37:35.635545 2964 raft_consensus.cc:482] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:35.635846 2964 raft_consensus.cc:504] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.637693 2964 leader_election.cc:288] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 753b64deae7e40f8ad4118c396ee192b (127.1.221.130:32911), 8fbad9813e3c452c986d66e41916f537 (127.1.221.129:36921)
I20260926 03:37:35.644516 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.638188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:32911 (local address 127.1.221.131:48903)
0926 03:37:35.638415 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.638445 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.638601 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.639055 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.639058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.639083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:35.639262 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.639274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.640512 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.640518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.642541 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.642554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.643815 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.643821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.644010 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.644045 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.644137 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.644279 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:37:35.644562 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.638160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:48903 (local address 127.1.221.130:32911)
0926 03:37:35.638410 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.638424 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.638525 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:37:35.638805 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.638812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.638924 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.639022 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:35.639427 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.640368 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.642704 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.643665 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.643784 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.643838 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.644171 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.644305 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:37:35.644884 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.638147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50257 (local address 127.1.221.129:36921)
0926 03:37:35.638351 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.638357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.639227 (+ 870us) server_negotiation.cc:407] Connection header received
0926 03:37:35.639360 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.639364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.639455 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.639522 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:35.640079 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.640957 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.643073 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.643989 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.644145 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.644213 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.644541 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.644680 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:37:35.645022 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.638188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36921 (local address 127.1.221.131:50257)
0926 03:37:35.639109 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:35.639129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:35.639242 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:35.639553 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:35.639557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:35.639580 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:35.639937 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.639949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.641100 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.641104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.642912 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:35.642926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.644186 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.644192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:35.644358 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.644410 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:37:35.644507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:37:35.644628 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":190,"thread_start_us":372,"threads_started":1}
I20260926 03:37:35.645900 2762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a670cf67d04baeb40f5b27b6f1e6c8" candidate_uuid: "3668ae5798f44fd082485506d3d627a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753b64deae7e40f8ad4118c396ee192b" is_pre_election: true
I20260926 03:37:35.645911 2628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a670cf67d04baeb40f5b27b6f1e6c8" candidate_uuid: "3668ae5798f44fd082485506d3d627a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbad9813e3c452c986d66e41916f537" is_pre_election: true
I20260926 03:37:35.646386 2762 raft_consensus.cc:2454] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3668ae5798f44fd082485506d3d627a0 in term 0.
I20260926 03:37:35.646435 2628 raft_consensus.cc:2454] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3668ae5798f44fd082485506d3d627a0 in term 0.
I20260926 03:37:35.647017 2831 leader_election.cc:302] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3668ae5798f44fd082485506d3d627a0, 8fbad9813e3c452c986d66e41916f537; no voters:
I20260926 03:37:35.647429 2964 raft_consensus.cc:2781] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:35.647567 2964 raft_consensus.cc:482] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:35.647663 2964 raft_consensus.cc:3037] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:35.649835 2964 raft_consensus.cc:504] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.650468 2964 leader_election.cc:288] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [CANDIDATE]: Term 1 election: Requested vote from peers 753b64deae7e40f8ad4118c396ee192b (127.1.221.130:32911), 8fbad9813e3c452c986d66e41916f537 (127.1.221.129:36921)
I20260926 03:37:35.650959 2762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a670cf67d04baeb40f5b27b6f1e6c8" candidate_uuid: "3668ae5798f44fd082485506d3d627a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753b64deae7e40f8ad4118c396ee192b"
I20260926 03:37:35.650972 2628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a670cf67d04baeb40f5b27b6f1e6c8" candidate_uuid: "3668ae5798f44fd082485506d3d627a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fbad9813e3c452c986d66e41916f537"
I20260926 03:37:35.651185 2762 raft_consensus.cc:3037] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:35.651185 2628 raft_consensus.cc:3037] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:37:35.652956 2941 tablet.cc:2369] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:35.654336 2628 raft_consensus.cc:2454] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3668ae5798f44fd082485506d3d627a0 in term 1.
I20260926 03:37:35.654780 2762 raft_consensus.cc:2454] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3668ae5798f44fd082485506d3d627a0 in term 1.
I20260926 03:37:35.654829 2831 leader_election.cc:302] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3668ae5798f44fd082485506d3d627a0, 8fbad9813e3c452c986d66e41916f537; no voters:
I20260926 03:37:35.655157 2964 raft_consensus.cc:2781] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:35.655632 2964 raft_consensus.cc:686] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [term 1 LEADER]: Becoming Leader. State: Replica: 3668ae5798f44fd082485506d3d627a0, State: Running, Role: LEADER
I20260926 03:37:35.656126 2964 consensus_queue.cc:237] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } }
I20260926 03:37:35.660672 2483 catalog_manager.cc:5495] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 3668ae5798f44fd082485506d3d627a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3668ae5798f44fd082485506d3d627a0 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "3668ae5798f44fd082485506d3d627a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753b64deae7e40f8ad4118c396ee192b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fbad9813e3c452c986d66e41916f537" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3668ae5798f44fd082485506d3d627a0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39515 } health_report { overall_health: HEALTHY } } }
W20260926 03:37:35.676573 2808 tablet.cc:2369] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 753b64deae7e40f8ad4118c396ee192b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:37:35.706678 2674 tablet.cc:2369] T f2a670cf67d04baeb40f5b27b6f1e6c8 P 8fbad9813e3c452c986d66e41916f537: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:35.714910 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.699632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.1.221.129:36921)
0926 03:37:35.699835 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.699840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.700401 (+ 561us) server_negotiation.cc:407] Connection header received
0926 03:37:35.700636 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.700641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.700709 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.700798 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.702262 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.703031 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.704334 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.704757 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.704900 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.709383 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.709397 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.709413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.709455 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.712135 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.712867 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.712875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.712878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.713021 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.713403 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.713407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.713409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.713776 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.714064 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.714351 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.714571 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:35.757658 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.743887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.1.221.129:36921)
0926 03:37:35.744103 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.744109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.744301 (+ 192us) server_negotiation.cc:407] Connection header received
0926 03:37:35.744484 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.744489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.744557 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.744645 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.745992 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.746725 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.748060 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.748473 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.748592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.752849 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.752855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.752859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.752895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.755435 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.756045 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.756048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.756051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.756179 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.756462 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.756464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.756467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.756724 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.756945 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.757176 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.757342 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:35.799084 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.785746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.1.221.129:36921)
0926 03:37:35.785985 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.785993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.786242 (+ 249us) server_negotiation.cc:407] Connection header received
0926 03:37:35.786406 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.786410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.786479 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.786560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.787891 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.788573 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.789688 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.790094 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.790181 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.793886 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.793890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.793894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.793928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.796874 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.797488 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.797491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.797494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.797609 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.797922 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.797925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.797927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.798214 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.798396 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.798641 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.798826 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:37:35.800882 2588 tablet_service.cc:2908] Rejecting scan request for tablet f2a670cf67d04baeb40f5b27b6f1e6c8: Uninitialized: clean time has not yet been initialized
I20260926 03:37:35.803646 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2545
I20260926 03:37:35.814908 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.804583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.1.221.129:36921)
0926 03:37:35.804741 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.804746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.805019 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:37:35.805147 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.805150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.805217 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.805293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.806343 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.807011 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.808170 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.808593 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.808691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.810628 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.810633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.810636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.810664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.812890 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.813498 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.813501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.813503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.813608 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.813875 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.813878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.813880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.814146 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.814341 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.814521 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.814689 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:37:35.815565 2668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:35.906080 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2545
I20260926 03:37:35.919939 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2679
Sep 26 03:37:35 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:37:35.936265 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:35.920784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.1.221.130:32911)
0926 03:37:35.920973 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:35.920978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:35.921032 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:37:35.921189 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:35.921194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:35.921259 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:35.921356 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:35.922521 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.923443 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.925202 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:35.925624 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:35.925713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:35.930450 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:35.930461 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:35.930475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:35.930517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:35.933484 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.934164 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.934171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.934173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.934326 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:35.934683 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:35.934687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:35.934689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:35.935105 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:35.935439 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:35.935725 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:37:35.935958 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:37:35.936885 2802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:36.022992 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2679
I20260926 03:37:36.035843 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2812
Sep 26 03:37:36 dist-test-slave-lqrk krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:37:36.051627 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:36.036561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.1.221.131:39515)
0926 03:37:36.037008 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:36.037013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:36.037030 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:37:36.037085 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:36.037089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:36.037167 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:36.037262 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:36.038437 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:36.039207 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:36.040782 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:36.041251 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:36.041377 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:36.045789 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:36.045806 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:36.045825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:36.045869 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:36.048763 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:36.049572 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:36.049579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:36.049582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:36.049709 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:36.050128 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:36.050132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:36.050134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:36.050534 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:36.050801 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:36.051052 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:37:36.051281 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260926 03:37:36.052318 2935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:36.140434 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2812
I20260926 03:37:36.140530 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:36.140039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:32911 (local address 127.1.221.131:43363)
0926 03:37:36.140209 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:36.140342 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.130:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:37:36.153141 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 2452
I20260926 03:37:36.154251 2514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:36.260771 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 2452
2026-09-26T03:37:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3925 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:36 dist-test-slave-lqrk krb5kdc[2988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:36 dist-test-slave-lqrk krb5kdc[2988](info): set up 2 sockets
Sep 26 03:37:36 dist-test-slave-lqrk krb5kdc[2988](info): commencing operation
krb5kdc: starting...
W20260926 03:37:37.251529 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:37 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:37.422354 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40903
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:37.734920 3001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:37.735201 3001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:37.735329 3001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:37.742588 3001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:37.742707 3001 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:37:37.742751 3001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:37.742834 3001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:37.742908 3001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:37.742991 3001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:37.752972 3001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:37.754266 3001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:37.755695 3001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:37.767728 3006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:37.767755 3007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:37.767817 3009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:37.769104 3001 server_base.cc:1034] running on GCE node
I20260926 03:37:37.769870 3001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:37.771310 3001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:37.772579 3001 hybrid_clock.cc:648] HybridClock initialized: now 1790393857772507 us; error 45 us; skew 500 ppm
I20260926 03:37:37.772909 3001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:37 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:37.775794 3001 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:37.777173 3001 webserver.cc:460] Webserver started at http://127.1.221.190:37127/ using document root <none> and password file <none>
I20260926 03:37:37.777853 3001 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:37.777953 3001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:37.778256 3001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:37.781191 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "64e6ebf9c33344bcbaa715140648cb82"
format_stamp: "Formatted at 2026-09-26 03:37:37 on dist-test-slave-lqrk"
server_key: "90894cd37bd4930f35a458cde8d1ee7b"
server_key_iv: "e5f6811dd3a9272de836ea58fd4e785e"
server_key_version: "encryptionkey@0"
I20260926 03:37:37.781961 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "64e6ebf9c33344bcbaa715140648cb82"
format_stamp: "Formatted at 2026-09-26 03:37:37 on dist-test-slave-lqrk"
server_key: "90894cd37bd4930f35a458cde8d1ee7b"
server_key_iv: "e5f6811dd3a9272de836ea58fd4e785e"
server_key_version: "encryptionkey@0"
I20260926 03:37:37.787133 3001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:37:37.790663 3016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:37.792107 3001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:37.792268 3001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "64e6ebf9c33344bcbaa715140648cb82"
format_stamp: "Formatted at 2026-09-26 03:37:37 on dist-test-slave-lqrk"
server_key: "90894cd37bd4930f35a458cde8d1ee7b"
server_key_iv: "e5f6811dd3a9272de836ea58fd4e785e"
server_key_version: "encryptionkey@0"
I20260926 03:37:37.792475 3001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:37.824347 3001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:37.828428 3001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:37.828675 3001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:37.849145 3001 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40903
I20260926 03:37:37.849162 3077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40903 every 8 connection(s)
I20260926 03:37:37.850521 3001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:37.854193 3078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:37.858103 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3001
I20260926 03:37:37.858333 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:37.859104 1910 external_mini_cluster.cc:1428] Setting key baa366f951feb9251f8e72e7c2fbc451
I20260926 03:37:37.865501 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Bootstrap starting.
I20260926 03:37:37.868474 3078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:37.869432 3078 log.cc:824] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:37.872602 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: No bootstrap required, opened a new log
Sep 26 03:37:37 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:37.881484 3078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } }
I20260926 03:37:37.881842 3078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:37.881891 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:37.861344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.1.221.190:40903)
0926 03:37:37.862145 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:37.862158 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:37.862204 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:37:37.864321 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:37.864345 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:37.864739 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:37.865106 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:37.866522 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:37.867892 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:37.869144 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:37.869910 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:37.870101 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:37.874621 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:37.874631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:37.874646 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:37.874680 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:37.877725 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:37.878523 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:37.878533 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:37.878539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:37.878718 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:37.879095 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:37.879098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:37.879100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:37.879724 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:37.879955 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:37.880390 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 03:37:37.880794 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":367,"threads_started":1}
I20260926 03:37:37.881951 3078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e6ebf9c33344bcbaa715140648cb82, State: Initialized, Role: FOLLOWER
I20260926 03:37:37.882550 3078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } }
I20260926 03:37:37.882789 3078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:37.882907 3078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:37.883066 3078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:37.885326 3078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } }
I20260926 03:37:37.885785 3078 leader_election.cc:302] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64e6ebf9c33344bcbaa715140648cb82; no voters:
I20260926 03:37:37.886286 3078 leader_election.cc:288] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:37.886406 3083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:37.887058 3083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [term 1 LEADER]: Becoming Leader. State: Replica: 64e6ebf9c33344bcbaa715140648cb82, State: Running, Role: LEADER
I20260926 03:37:37.887633 3083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } }
I20260926 03:37:37.888247 3078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:37.890071 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64e6ebf9c33344bcbaa715140648cb82. Latest consensus state: current_term: 1 leader_uuid: "64e6ebf9c33344bcbaa715140648cb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } } }
I20260926 03:37:37.890465 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:37.892041 3088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:37.892019 3084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64e6ebf9c33344bcbaa715140648cb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6ebf9c33344bcbaa715140648cb82" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40903 } } }
I20260926 03:37:37.892283 3084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:37.897030 3088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:37.914065 3088 catalog_manager.cc:1347] Generated new cluster ID: f5f859d2d67b48bbb3c05db90ab4373c
I20260926 03:37:37.914196 3088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:37.921581 3088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:37.923468 3088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:37.945571 3088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Generated new TSK 0
I20260926 03:37:37.946460 3088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:37.994837 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40903
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:38.314509 3104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:38.314798 3104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:38.314931 3104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:38.322441 3104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:38.322557 3104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:38.322666 3104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:38.322760 3104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:38.332723 3104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:40903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:38.333998 3104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:38.335350 3104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:38.348029 3112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:38.348127 3110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:38.348613 3109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:38.348846 3104 server_base.cc:1034] running on GCE node
I20260926 03:37:38.349457 3104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:38.350359 3104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:38.351553 3104 hybrid_clock.cc:648] HybridClock initialized: now 1790393858351490 us; error 35 us; skew 500 ppm
I20260926 03:37:38.351840 3104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:38 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393858, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:38.354835 3104 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:38.356431 3104 webserver.cc:460] Webserver started at http://127.1.221.129:45427/ using document root <none> and password file <none>
I20260926 03:37:38.357105 3104 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:38.357192 3104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:38.357450 3104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:38.360240 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "ea5eb6894c064e828ea7129c4ae41752"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "6c22f1a2bca0c8f02da1a3d0b6134f26"
server_key_iv: "892504d09e9826366531ecd225b1f961"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.360998 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "ea5eb6894c064e828ea7129c4ae41752"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "6c22f1a2bca0c8f02da1a3d0b6134f26"
server_key_iv: "892504d09e9826366531ecd225b1f961"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.365978 3104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:38.369329 3119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:38.370616 3104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:38.370754 3104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "ea5eb6894c064e828ea7129c4ae41752"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "6c22f1a2bca0c8f02da1a3d0b6134f26"
server_key_iv: "892504d09e9826366531ecd225b1f961"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.370965 3104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:38.395545 3104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:38.399878 3104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:38.400126 3104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:38.402810 3104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:38.402892 3104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:38.403021 3104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:38.403131 3104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 03:37:38 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393858, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:38.425910 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:48601 (local address 127.1.221.190:40903)
0926 03:37:38.405896 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:38.405901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:38.405920 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:37:38.408022 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:38.408028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:38.408119 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:38.408245 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:38.410751 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.411907 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.414094 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.414654 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.414774 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.419012 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:38.419017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:38.419020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:38.419054 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:38.421964 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.422951 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.422957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.422960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.423108 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.423715 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.423720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.423723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.424101 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:38.424255 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:38.425256 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0926 03:37:38.425504 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":2433,"threads_started":1}
I20260926 03:37:38.427407 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.404075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.129:48601)
0926 03:37:38.405061 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:38.405109 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:38.407660 (+ 2551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:38.408562 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:38.408574 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:38.409489 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:38.410495 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.410532 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.412084 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.412089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.413900 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.413915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.414912 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.414920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.415128 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.415907 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:38.415933 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:38.418771 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:38.422105 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.422115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.422140 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.422754 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.423230 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.423235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.423239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.423567 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.424249 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:38.424259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:38.424767 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 03:37:38.425118 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 03:37:38.425726 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":380,"threads_started":1}
I20260926 03:37:38.455240 3104 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:37161
I20260926 03:37:38.455256 3239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:37161 every 8 connection(s)
I20260926 03:37:38.456240 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.435017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38703 (local address 127.1.221.190:40903)
0926 03:37:38.435234 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:38.435240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:38.441967 (+ 6727us) server_negotiation.cc:407] Connection header received
0926 03:37:38.442164 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:38.442195 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:38.442332 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:38.442498 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:38.444375 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.445264 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.447448 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.447971 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.448070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.450554 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:38.450561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:38.450566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:38.450607 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:38.453477 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.454344 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.454351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.454355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.454514 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.454938 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.454941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.454943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.455226 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:38.455335 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:38.455834 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 03:37:38.456018 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:37:38.456483 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.434970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.129:38703)
0926 03:37:38.441822 (+ 6852us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:38.441843 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:38.442019 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:38.442801 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:38.442807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:38.443693 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:38.444191 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.444206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.445441 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.445447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.447221 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.447238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.448167 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.448174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.448398 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.449077 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:38.449102 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:38.450344 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:38.453603 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.453609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.453613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.454203 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.454651 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.454656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.454660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.454827 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.455325 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:38.455330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:38.455566 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 03:37:38.455733 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 03:37:38.456052 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6688,"thread_start_us":6482,"threads_started":1}
I20260926 03:37:38.457007 3104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:38.459154 3240 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40903
I20260926 03:37:38.459398 3240 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:38.460186 3240 heartbeater.cc:511] Master 127.1.221.190:40903 requested a full tablet report, sending...
I20260926 03:37:38.462472 3033 ts_manager.cc:194] Registered new tserver with Master: ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161)
I20260926 03:37:38.463887 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3104
I20260926 03:37:38.464099 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:38.464710 1910 external_mini_cluster.cc:1428] Setting key 4608db88968ae2da078b89fa9c39650c
I20260926 03:37:38.466133 3033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:48601
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:38.503302 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40903
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:38.814265 3244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:38.814545 3244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:38.814674 3244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:38.822315 3244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:38.822438 3244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:38.822567 3244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:38.822677 3244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:38.832317 3244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:40903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:38.833557 3244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:38.834865 3244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:38.847640 3250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:38.847608 3252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:38.847751 3249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:38.848366 3244 server_base.cc:1034] running on GCE node
I20260926 03:37:38.848971 3244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:38.849967 3244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:38.851311 3244 hybrid_clock.cc:648] HybridClock initialized: now 1790393858851236 us; error 46 us; skew 500 ppm
I20260926 03:37:38.851661 3244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:38 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393858, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:38.859134 3244 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:38.860730 3244 webserver.cc:460] Webserver started at http://127.1.221.130:42101/ using document root <none> and password file <none>
I20260926 03:37:38.861418 3244 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:38.861506 3244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:38.861819 3244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:38.864681 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "96cfbd44057f455eb9c0a5bb415a236d"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "d735298986ffd850c48796ed68aee07f"
server_key_iv: "fbbd0678d3d35aaa234e3234a0c6ed27"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.865444 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "96cfbd44057f455eb9c0a5bb415a236d"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "d735298986ffd850c48796ed68aee07f"
server_key_iv: "fbbd0678d3d35aaa234e3234a0c6ed27"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.870641 3244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:37:38.874222 3259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:38.875708 3244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:38.875857 3244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "96cfbd44057f455eb9c0a5bb415a236d"
format_stamp: "Formatted at 2026-09-26 03:37:38 on dist-test-slave-lqrk"
server_key: "d735298986ffd850c48796ed68aee07f"
server_key_iv: "fbbd0678d3d35aaa234e3234a0c6ed27"
server_key_version: "encryptionkey@0"
I20260926 03:37:38.876060 3244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:38.893296 3244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:38.897672 3244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:38.897927 3244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:38.900655 3244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:38.900741 3244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:38.900870 3244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:38.900938 3244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:37:38 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393858, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:38.924530 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.901712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49099 (local address 127.1.221.190:40903)
0926 03:37:38.901901 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:38.901905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:38.903266 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 03:37:38.905886 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:38.905892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:38.905993 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:38.906110 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:38.908473 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.909640 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.912060 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.912640 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.912765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.917474 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:38.917481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:38.917487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:38.917533 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:38.920876 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.921776 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.921782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.921785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.921948 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.922493 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.922499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.922502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.922848 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:38.923001 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:38.924096 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 03:37:38.924276 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:37:38.926548 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.902185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.130:49099)
0926 03:37:38.903089 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:38.903128 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:38.905458 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:38.906427 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:38.906444 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:38.907242 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:38.908262 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.908285 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.909808 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.909813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.911853 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.911869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.913096 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.913105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.913402 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.914203 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:38.914226 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:38.917181 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:38.921011 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.921019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.921042 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.921644 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.922079 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.922084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.922088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.922320 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.922984 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:38.922992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:38.923564 (+ 572us) client_negotiation.cc:769] Sending connection context
0926 03:37:38.923966 (+ 402us) client_negotiation.cc:240] Negotiation successful
0926 03:37:38.924615 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":409,"threads_started":1}
I20260926 03:37:38.942139 3244 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:35709
I20260926 03:37:38.942143 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:35709 every 8 connection(s)
I20260926 03:37:38.943773 3244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:38.946306 3379 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40903
I20260926 03:37:38.946559 3379 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:38.947201 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.932369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55271 (local address 127.1.221.190:40903)
0926 03:37:38.932533 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:38.932540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:38.933483 (+ 943us) server_negotiation.cc:407] Connection header received
0926 03:37:38.933630 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:38.933636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:38.933719 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:38.933835 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:38.935041 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.936039 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.937953 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.938505 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.938634 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.940823 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:38.940830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:38.940835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:38.940874 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:38.944224 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.945117 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.945123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.945126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.945286 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:38.945691 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:38.945696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:38.945699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:38.946143 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:38.946281 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:38.946800 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 03:37:38.946974 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:37:38.947415 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:38.932699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.130:55271)
0926 03:37:38.933364 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:38.933384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:38.933515 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:38.933877 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:38.933881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:38.934426 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:38.934847 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.934865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.936177 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.936182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.937747 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:38.937762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:38.938751 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:38.938759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:38.938948 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:38.939691 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:38.939712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:38.940627 (+ 915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:38.944377 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.944387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.944391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.944984 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.945414 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:38.945418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:38.945421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:38.945551 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:38.946245 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:38.946250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:38.946457 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 03:37:38.946617 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:37:38.946953 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":370,"threads_started":1}
I20260926 03:37:38.947436 3379 heartbeater.cc:511] Master 127.1.221.190:40903 requested a full tablet report, sending...
I20260926 03:37:38.948838 3033 ts_manager.cc:194] Registered new tserver with Master: 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130:35709)
I20260926 03:37:38.949683 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3244
I20260926 03:37:38.949718 3033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:49099
I20260926 03:37:38.949874 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:38.950423 1910 external_mini_cluster.cc:1428] Setting key fd1f03a3acd5f27aeeadbcc74284ca55
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:38.990567 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40903
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:39.302316 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:39.302632 3383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:39.302758 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:39.310218 3383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:39.310357 3383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:39.310482 3383 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:39.310593 3383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:39.320535 3383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:40903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:39.321828 3383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:39.323184 3383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:39.335716 3391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:39.335775 3388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:39.335752 3389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:39.336891 3383 server_base.cc:1034] running on GCE node
I20260926 03:37:39.337472 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:39.338325 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:39.339488 3383 hybrid_clock.cc:648] HybridClock initialized: now 1790393859339425 us; error 32 us; skew 500 ppm
I20260926 03:37:39.339767 3383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:39 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:39.342664 3383 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:39.344066 3383 webserver.cc:460] Webserver started at http://127.1.221.131:44979/ using document root <none> and password file <none>
I20260926 03:37:39.344745 3383 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:39.344827 3383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:39.345115 3383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:39.347922 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "77c67aa3c91a443793ecd8aa31bebe69"
format_stamp: "Formatted at 2026-09-26 03:37:39 on dist-test-slave-lqrk"
server_key: "d2ac50fee385cf776a86ad05993530ef"
server_key_iv: "d6c856cbdd7fd4cb560e28587244a078"
server_key_version: "encryptionkey@0"
I20260926 03:37:39.348640 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "77c67aa3c91a443793ecd8aa31bebe69"
format_stamp: "Formatted at 2026-09-26 03:37:39 on dist-test-slave-lqrk"
server_key: "d2ac50fee385cf776a86ad05993530ef"
server_key_iv: "d6c856cbdd7fd4cb560e28587244a078"
server_key_version: "encryptionkey@0"
I20260926 03:37:39.353398 3383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:37:39.356587 3398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:39.357834 3383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:39.357963 3383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "77c67aa3c91a443793ecd8aa31bebe69"
format_stamp: "Formatted at 2026-09-26 03:37:39 on dist-test-slave-lqrk"
server_key: "d2ac50fee385cf776a86ad05993530ef"
server_key_iv: "d6c856cbdd7fd4cb560e28587244a078"
server_key_version: "encryptionkey@0"
I20260926 03:37:39.358142 3383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:39.392598 3383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:39.396772 3383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:39.397008 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:39.399672 3383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:39.399771 3383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:39.399909 3383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:39.399973 3383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:37:39 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:39.423359 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.400303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53945 (local address 127.1.221.190:40903)
0926 03:37:39.400484 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.400488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.402110 (+ 1622us) server_negotiation.cc:407] Connection header received
0926 03:37:39.404233 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.404240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.404338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.404423 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:39.406411 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.407465 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.410474 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.410991 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.411113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.416149 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:39.416158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:39.416164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:39.416245 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:39.419754 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.420853 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.420859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.420863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.421009 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.421487 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.421493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.421497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.421794 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:39.421946 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.422758 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.423006 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:37:39.424933 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.401128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.131:53945)
0926 03:37:39.401951 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.401989 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.403928 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.404707 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.404716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.405356 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:39.406207 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.406233 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.407666 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.407672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.410287 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.410319 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.411234 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.411242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.411627 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.412454 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:39.412492 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:39.415753 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:39.419964 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:39.419975 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:39.420003 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:39.420709 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:39.421132 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:39.421136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:39.421138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:39.421363 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:39.421940 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:39.421948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:39.422346 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.422598 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.423170 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260926 03:37:39.444594 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.431597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.1.221.131:59465)
0926 03:37:39.432153 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.432169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.432296 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.432663 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.432667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.433118 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:39.433393 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.433402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.434398 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.434403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.435457 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.435470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.436371 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.436378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.436589 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.437208 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:39.437231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:39.438322 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:39.441568 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:39.441573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:39.441577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:39.442154 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:39.442571 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:39.442576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:39.442580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:39.442730 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:39.443752 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:39.443757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:39.443918 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.444037 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.444278 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":307,"threads_started":1}
I20260926 03:37:39.444828 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.431531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59465 (local address 127.1.221.190:40903)
0926 03:37:39.431682 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.431687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.432257 (+ 570us) server_negotiation.cc:407] Connection header received
0926 03:37:39.432430 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.432436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.432534 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.432623 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:39.433539 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.434244 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.435586 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.436119 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.436250 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.438508 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:39.438512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:39.438516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:39.438546 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:39.441429 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.442285 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.442290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.442294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.442470 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.443190 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.443196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.443200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.443622 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:39.443790 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.444181 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.444443 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:37:39.455657 3383 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36673
I20260926 03:37:39.455725 3517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36673 every 8 connection(s)
I20260926 03:37:39.456866 3383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:39.457947 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3383
I20260926 03:37:39.458132 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:39.458637 1910 external_mini_cluster.cc:1428] Setting key f8867ad4c9afe55d40ac872fb31f1ac5
I20260926 03:37:39.460764 3518 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40903
I20260926 03:37:39.462181 3518 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:39.462860 3518 heartbeater.cc:511] Master 127.1.221.190:40903 requested a full tablet report, sending...
I20260926 03:37:39.464103 3033 ts_manager.cc:194] Registered new tserver with Master: 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:39.464927 3033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:53945
I20260926 03:37:39.468947 3240 heartbeater.cc:503] Master 127.1.221.190:40903 was elected leader, sending a full tablet report...
I20260926 03:37:39.473992 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:37:39 dist-test-slave-lqrk krb5kdc[2988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:37:39 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:39.515157 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.502686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.1.221.190:40903)
0926 03:37:39.502865 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.502869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.503208 (+ 339us) server_negotiation.cc:407] Connection header received
0926 03:37:39.503361 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.503365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.503459 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.503552 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:39.504876 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.505606 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.506787 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.507171 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.507257 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.510861 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:39.510866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:39.510869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:39.510901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:39.513142 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.513710 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.513714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.513716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.513827 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:39.514115 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:39.514118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:39.514120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:39.514392 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:39.514560 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.514746 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.514902 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:37:39.519155 3033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:39.525774 3033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:39.548894 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.537623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35709 (local address 127.0.0.1:40338)
0926 03:37:39.539829 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.539843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.539934 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.540934 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.540942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.540971 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:39.541408 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.541421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.543146 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.543151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.546092 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.546106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.547895 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.547901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.548088 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.548140 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.548260 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.548460 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":1958,"threads_started":1}
I20260926 03:37:39.549640 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.537512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37161 (local address 127.0.0.1:53058)
0926 03:37:39.539699 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.539751 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.539931 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.541980 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.541985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.542009 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:39.542378 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.542389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.544129 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.544134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.547035 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.547046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.548853 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.548860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.549039 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.549071 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.549159 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.549276 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"mutex_wait_us":1,"thread_start_us":1803,"threads_started":1}
I20260926 03:37:39.549777 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.538000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.1.221.130:35709)
0926 03:37:39.539421 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.539429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.540112 (+ 683us) server_negotiation.cc:407] Connection header received
0926 03:37:39.540216 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.540224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.540511 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.540686 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:39.541611 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.543002 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.546266 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.547722 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.547862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.548069 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.549327 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.549502 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":1121,"threads_started":1}
I20260926 03:37:39.550578 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.537670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.1.221.129:37161)
0926 03:37:39.539086 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.539096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.540114 (+ 1018us) server_negotiation.cc:407] Connection header received
0926 03:37:39.540216 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.540224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.540505 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.540687 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:39.542559 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.543973 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.547193 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.548631 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.548760 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.548956 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.550141 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.550305 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":1135,"threads_started":1}
I20260926 03:37:39.556082 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.540241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36673 (local address 127.0.0.1:52050)
0926 03:37:39.547520 (+ 7279us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.547538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.547654 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.548292 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.548296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.548319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:39.548666 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.548676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.550837 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.550843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.552818 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.552833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.555303 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.555308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.555509 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.555540 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.555622 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.555749 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7165,"thread_start_us":7018,"threads_started":1}
I20260926 03:37:39.555665 3170 tablet_service.cc:1467] Processing CreateTablet for tablet 854f5cb8d3104af0a025dc3607436066 (DEFAULT_TABLE table=test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.555718 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 854f5cb8d3104af0a025dc3607436066 (DEFAULT_TABLE table=test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.557087 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.540663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.1.221.131:36673)
0926 03:37:39.545177 (+ 4514us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.545186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.547651 (+ 2465us) server_negotiation.cc:407] Connection header received
0926 03:37:39.547808 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.547816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.548095 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.548294 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:39.548898 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.550668 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.553012 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.554657 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.554934 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.555160 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.556589 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.556756 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4348,"thread_start_us":4189,"threads_started":1}
I20260926 03:37:39.557109 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854f5cb8d3104af0a025dc3607436066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.557125 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854f5cb8d3104af0a025dc3607436066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.560415 3448 tablet_service.cc:1467] Processing CreateTablet for tablet 854f5cb8d3104af0a025dc3607436066 (DEFAULT_TABLE table=test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.562076 3448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 854f5cb8d3104af0a025dc3607436066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.571830 3537 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Bootstrap starting.
I20260926 03:37:39.576162 3537 tablet_bootstrap.cc:654] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.577368 3538 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Bootstrap starting.
I20260926 03:37:39.579244 3537 log.cc:824] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:39.581373 3538 tablet_bootstrap.cc:654] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.582491 3537 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: No bootstrap required, opened a new log
I20260926 03:37:39.582787 3537 ts_tablet_manager.cc:1397] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 03:37:39.583134 3538 log.cc:824] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:39.580644 3539 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Bootstrap starting.
I20260926 03:37:39.586537 3538 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: No bootstrap required, opened a new log
I20260926 03:37:39.586802 3538 ts_tablet_manager.cc:1397] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:37:39.590692 3537 raft_consensus.cc:348] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.591061 3537 raft_consensus.cc:374] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.591161 3537 raft_consensus.cc:729] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5eb6894c064e828ea7129c4ae41752, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.591747 3537 consensus_queue.cc:260] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.593302 3537 ts_tablet_manager.cc:1428] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260926 03:37:39.597370 3539 tablet_bootstrap.cc:654] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.598691 3539 log.cc:824] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:39.602483 3539 tablet_bootstrap.cc:492] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: No bootstrap required, opened a new log
I20260926 03:37:39.602787 3539 ts_tablet_manager.cc:1397] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260926 03:37:39.602241 3538 raft_consensus.cc:348] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.602991 3538 raft_consensus.cc:374] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.603133 3538 raft_consensus.cc:729] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c67aa3c91a443793ecd8aa31bebe69, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.603894 3538 consensus_queue.cc:260] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.605042 3518 heartbeater.cc:503] Master 127.1.221.190:40903 was elected leader, sending a full tablet report...
I20260926 03:37:39.605783 3538 ts_tablet_manager.cc:1428] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Time spent starting tablet: real 0.019s user 0.014s sys 0.000s
I20260926 03:37:39.612430 3539 raft_consensus.cc:348] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.612922 3539 raft_consensus.cc:374] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.613019 3539 raft_consensus.cc:729] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cfbd44057f455eb9c0a5bb415a236d, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.613620 3539 consensus_queue.cc:260] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.614737 3379 heartbeater.cc:503] Master 127.1.221.190:40903 was elected leader, sending a full tablet report...
I20260926 03:37:39.615356 3539 ts_tablet_manager.cc:1428] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260926 03:37:39.696429 3380 tablet.cc:2369] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:37:39.711483 3519 tablet.cc:2369] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:37:39.714973 3241 tablet.cc:2369] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:39.785537 3545 raft_consensus.cc:482] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:39.785838 3545 raft_consensus.cc:504] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.787465 3545 leader_election.cc:288] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:39.794293 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.787889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:48115 (local address 127.1.221.129:37161)
0926 03:37:39.788076 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.788080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.788466 (+ 386us) server_negotiation.cc:407] Connection header received
0926 03:37:39.788800 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.788807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.788885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.788949 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:39.789666 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.790381 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.792608 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.793426 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.793582 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.793659 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.793934 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.794061 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:37:39.794540 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.788637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51095 (local address 127.1.221.131:36673)
0926 03:37:39.788834 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.788839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.789207 (+ 368us) server_negotiation.cc:407] Connection header received
0926 03:37:39.789386 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.789390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.789450 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.789543 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:39.790111 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.790830 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.792853 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.793654 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.793798 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.793867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.794168 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.794298 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:37:39.794670 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.788461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36673 (local address 127.1.221.130:51095)
0926 03:37:39.789078 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.789095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.789240 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.789557 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.789561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.789583 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:39.789959 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.789969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.790977 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.790981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.792699 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.792710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.793875 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.793881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.794011 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.794034 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.794121 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.794307 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":25,"thread_start_us":333,"threads_started":1}
I20260926 03:37:39.794840 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.787710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37161 (local address 127.1.221.130:48115)
0926 03:37:39.788365 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:39.788384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:39.788590 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:39.789126 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:39.789129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:39.789158 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:39.789480 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.789488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.790518 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.790523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.792449 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:39.792465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.793625 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.793631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:39.793772 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.793797 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:37:39.793890 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:37:39.794024 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1,"spinlock_wait_cycles":17792,"thread_start_us":311,"threads_started":1}
I20260926 03:37:39.795650 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854f5cb8d3104af0a025dc3607436066" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69" is_pre_election: true
I20260926 03:37:39.795770 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854f5cb8d3104af0a025dc3607436066" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5eb6894c064e828ea7129c4ae41752" is_pre_election: true
I20260926 03:37:39.796178 3468 raft_consensus.cc:2454] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 0.
I20260926 03:37:39.796219 3190 raft_consensus.cc:2454] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 0.
I20260926 03:37:39.796823 3261 leader_election.cc:302] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cfbd44057f455eb9c0a5bb415a236d, ea5eb6894c064e828ea7129c4ae41752; no voters:
I20260926 03:37:39.797247 3545 raft_consensus.cc:2781] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:39.797394 3545 raft_consensus.cc:482] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:39.797506 3545 raft_consensus.cc:3037] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:39.799980 3545 raft_consensus.cc:504] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.800659 3545 leader_election.cc:288] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 election: Requested vote from peers ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:39.801209 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854f5cb8d3104af0a025dc3607436066" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5eb6894c064e828ea7129c4ae41752"
I20260926 03:37:39.801256 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "854f5cb8d3104af0a025dc3607436066" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69"
I20260926 03:37:39.801410 3190 raft_consensus.cc:3037] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:39.801442 3468 raft_consensus.cc:3037] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:39.803716 3468 raft_consensus.cc:2454] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 1.
I20260926 03:37:39.803820 3190 raft_consensus.cc:2454] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 1.
I20260926 03:37:39.804167 3261 leader_election.cc:302] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c67aa3c91a443793ecd8aa31bebe69, 96cfbd44057f455eb9c0a5bb415a236d; no voters:
I20260926 03:37:39.804551 3545 raft_consensus.cc:2781] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:39.805001 3545 raft_consensus.cc:686] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 1 LEADER]: Becoming Leader. State: Replica: 96cfbd44057f455eb9c0a5bb415a236d, State: Running, Role: LEADER
I20260926 03:37:39.805713 3545 consensus_queue.cc:237] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.810685 3033 catalog_manager.cc:5495] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130). New cstate: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } health_report { overall_health: UNKNOWN } } }
I20260926 03:37:39.850114 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:39.844607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.1.221.130:35709)
0926 03:37:39.844823 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:39.844829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:39.844968 (+ 139us) server_negotiation.cc:407] Connection header received
0926 03:37:39.845191 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:39.845195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:39.845282 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:39.845406 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:39.845756 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.846490 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.847845 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:39.848247 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:39.848345 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:39.848869 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:39.849544 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:39.849669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:39.849768 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:37:39.849851 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:37:39.859135 3190 raft_consensus.cc:1261] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Refusing update from remote peer 96cfbd44057f455eb9c0a5bb415a236d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:39.859135 3468 raft_consensus.cc:1261] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Refusing update from remote peer 96cfbd44057f455eb9c0a5bb415a236d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:39.860287 3548 consensus_queue.cc:1035] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:39.860728 3545 consensus_queue.cc:1035] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:39.872782 3551 mvcc.cc:205] Tried to move back new op lower bound from 7333453249971695616 to 7333453249768189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333453249971695616}]
I20260926 03:37:39.877708 3553 mvcc.cc:205] Tried to move back new op lower bound from 7333453249971695616 to 7333453249768189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333453249971695616}]
I20260926 03:37:39.892690 3032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:39.893010 3032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:39.896921 3032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Sending DeleteTablet for 3 replicas of tablet 854f5cb8d3104af0a025dc3607436066
I20260926 03:37:39.898211 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet 854f5cb8d3104af0a025dc3607436066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40338
I20260926 03:37:39.898454 3170 tablet_service.cc:1514] Processing DeleteTablet for tablet 854f5cb8d3104af0a025dc3607436066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53058
I20260926 03:37:39.899133 3448 tablet_service.cc:1514] Processing DeleteTablet for tablet 854f5cb8d3104af0a025dc3607436066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:52050
I20260926 03:37:39.899765 3032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:39.900621 3032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:39.903723 3563 tablet_replica.cc:330] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: stopping tablet replica
I20260926 03:37:39.904242 3563 raft_consensus.cc:2229] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:39.904776 3563 raft_consensus.cc:2258] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:39.905548 3562 tablet_replica.cc:330] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: stopping tablet replica
I20260926 03:37:39.906000 3562 raft_consensus.cc:2229] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:39.906489 3562 raft_consensus.cc:2258] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:39.907253 3561 tablet_replica.cc:330] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: stopping tablet replica
I20260926 03:37:39.907677 3561 raft_consensus.cc:2229] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:39.908690 3562 ts_tablet_manager.cc:1905] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:39.909657 3448 tablet_service.cc:1467] Processing CreateTablet for tablet 01d6f10f70804615be1b3ad80c789c9d (DEFAULT_TABLE table=test-table [id=15642cc3fcbc4ad58951941a0154cbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.910117 3448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d6f10f70804615be1b3ad80c789c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.910337 3563 ts_tablet_manager.cc:1905] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:39.911623 3170 tablet_service.cc:1467] Processing CreateTablet for tablet 01d6f10f70804615be1b3ad80c789c9d (DEFAULT_TABLE table=test-table [id=15642cc3fcbc4ad58951941a0154cbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.911818 3561 raft_consensus.cc:2258] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:39.912081 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d6f10f70804615be1b3ad80c789c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.915330 3561 ts_tablet_manager.cc:1905] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:39.917330 3563 ts_tablet_manager.cc:1918] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:37:39.917502 3563 log.cc:1196] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/854f5cb8d3104af0a025dc3607436066
I20260926 03:37:39.917982 3563 ts_tablet_manager.cc:1939] T 854f5cb8d3104af0a025dc3607436066 P 77c67aa3c91a443793ecd8aa31bebe69: Deleting consensus metadata
I20260926 03:37:39.917900 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 01d6f10f70804615be1b3ad80c789c9d (DEFAULT_TABLE table=test-table [id=15642cc3fcbc4ad58951941a0154cbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:39.918445 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d6f10f70804615be1b3ad80c789c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:39.919950 3018 catalog_manager.cc:4841] TS 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673): tablet 854f5cb8d3104af0a025dc3607436066 (table test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]) successfully deleted
I20260926 03:37:39.922139 3562 ts_tablet_manager.cc:1918] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:37:39.922305 3562 log.cc:1196] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/854f5cb8d3104af0a025dc3607436066
I20260926 03:37:39.922559 3561 ts_tablet_manager.cc:1918] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:37:39.922729 3561 log.cc:1196] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/854f5cb8d3104af0a025dc3607436066
I20260926 03:37:39.922828 3562 ts_tablet_manager.cc:1939] T 854f5cb8d3104af0a025dc3607436066 P ea5eb6894c064e828ea7129c4ae41752: Deleting consensus metadata
I20260926 03:37:39.923231 3561 ts_tablet_manager.cc:1939] T 854f5cb8d3104af0a025dc3607436066 P 96cfbd44057f455eb9c0a5bb415a236d: Deleting consensus metadata
I20260926 03:37:39.925077 3017 catalog_manager.cc:4841] TS 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130:35709): tablet 854f5cb8d3104af0a025dc3607436066 (table test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]) successfully deleted
I20260926 03:37:39.928318 3018 catalog_manager.cc:4841] TS ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161): tablet 854f5cb8d3104af0a025dc3607436066 (table test-table [id=5ed53492de764ea0bb6dbb1ef0a72369]) successfully deleted
I20260926 03:37:39.929097 3538 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Bootstrap starting.
I20260926 03:37:39.929129 3537 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Bootstrap starting.
I20260926 03:37:39.930296 3539 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Bootstrap starting.
I20260926 03:37:39.930706 3537 tablet_bootstrap.cc:654] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.930727 3538 tablet_bootstrap.cc:654] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.931900 3539 tablet_bootstrap.cc:654] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:39.934926 3539 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: No bootstrap required, opened a new log
I20260926 03:37:39.935070 3539 ts_tablet_manager.cc:1397] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 03:37:39.935334 3537 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: No bootstrap required, opened a new log
I20260926 03:37:39.935514 3537 ts_tablet_manager.cc:1397] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260926 03:37:39.936036 3538 tablet_bootstrap.cc:492] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: No bootstrap required, opened a new log
I20260926 03:37:39.935914 3539 raft_consensus.cc:348] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.936136 3539 raft_consensus.cc:374] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.936239 3539 raft_consensus.cc:729] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cfbd44057f455eb9c0a5bb415a236d, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.936352 3538 ts_tablet_manager.cc:1397] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 03:37:39.936246 3537 raft_consensus.cc:348] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.936460 3537 raft_consensus.cc:374] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.936550 3537 raft_consensus.cc:729] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5eb6894c064e828ea7129c4ae41752, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.936494 3539 consensus_queue.cc:260] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.936841 3537 consensus_queue.cc:260] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.937078 3539 ts_tablet_manager.cc:1428] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:37:39.938297 3537 ts_tablet_manager.cc:1428] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:37:39.937448 3538 raft_consensus.cc:348] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.939599 3538 raft_consensus.cc:374] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:39.939692 3538 raft_consensus.cc:729] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c67aa3c91a443793ecd8aa31bebe69, State: Initialized, Role: FOLLOWER
I20260926 03:37:39.940127 3538 consensus_queue.cc:260] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:39.940865 3538 ts_tablet_manager.cc:1428] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 03:37:40.016439 3542 raft_consensus.cc:482] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:40.016685 3542 raft_consensus.cc:504] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.018612 3542 leader_election.cc:288] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130:35709), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:40.026993 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.019032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:57389 (local address 127.1.221.130:35709)
0926 03:37:40.019219 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.019225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.020047 (+ 822us) server_negotiation.cc:407] Connection header received
0926 03:37:40.020234 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.020239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.020335 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.020446 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:40.021088 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.022244 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.024995 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.025802 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.025975 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.026056 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.026398 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.026584 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:37:40.027081 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.019078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:48939 (local address 127.1.221.131:36673)
0926 03:37:40.019261 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.019268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.019927 (+ 659us) server_negotiation.cc:407] Connection header received
0926 03:37:40.020147 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.020152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.020229 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.020334 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:40.021049 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.021775 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.024961 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.026141 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.026300 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.026377 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.026698 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.026837 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:37:40.027144 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.018826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36673 (local address 127.1.221.129:48939)
0926 03:37:40.019789 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.019822 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.019968 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.020369 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.020374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.020414 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:40.020824 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.020844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.021915 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.021920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.024790 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.024803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.026365 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.026371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.026539 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.026570 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.026648 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.026783 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":59,"thread_start_us":785,"threads_started":1}
I20260926 03:37:40.027804 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.018840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35709 (local address 127.1.221.129:57389)
0926 03:37:40.019919 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.019937 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.020078 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.020592 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.020595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.020613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:40.020941 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.020950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.022427 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.022432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.024788 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.024803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.026024 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.026030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.026189 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.026215 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.027204 (+ 989us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.027515 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":164,"thread_start_us":482,"threads_started":1}
I20260926 03:37:40.028002 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d6f10f70804615be1b3ad80c789c9d" candidate_uuid: "ea5eb6894c064e828ea7129c4ae41752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69" is_pre_election: true
I20260926 03:37:40.028266 3468 raft_consensus.cc:2454] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5eb6894c064e828ea7129c4ae41752 in term 0.
I20260926 03:37:40.028774 3121 leader_election.cc:302] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c67aa3c91a443793ecd8aa31bebe69, ea5eb6894c064e828ea7129c4ae41752; no voters:
I20260926 03:37:40.028795 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d6f10f70804615be1b3ad80c789c9d" candidate_uuid: "ea5eb6894c064e828ea7129c4ae41752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cfbd44057f455eb9c0a5bb415a236d" is_pre_election: true
I20260926 03:37:40.029244 3542 raft_consensus.cc:2781] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:40.029254 3329 raft_consensus.cc:2454] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5eb6894c064e828ea7129c4ae41752 in term 0.
I20260926 03:37:40.029366 3542 raft_consensus.cc:482] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:40.029453 3542 raft_consensus.cc:3037] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.032667 3542 raft_consensus.cc:504] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.033371 3542 leader_election.cc:288] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [CANDIDATE]: Term 1 election: Requested vote from peers 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130:35709), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:40.033895 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d6f10f70804615be1b3ad80c789c9d" candidate_uuid: "ea5eb6894c064e828ea7129c4ae41752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69"
I20260926 03:37:40.033900 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d6f10f70804615be1b3ad80c789c9d" candidate_uuid: "ea5eb6894c064e828ea7129c4ae41752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cfbd44057f455eb9c0a5bb415a236d"
I20260926 03:37:40.034119 3468 raft_consensus.cc:3037] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.034125 3329 raft_consensus.cc:3037] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.037496 3468 raft_consensus.cc:2454] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5eb6894c064e828ea7129c4ae41752 in term 1.
I20260926 03:37:40.037506 3329 raft_consensus.cc:2454] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5eb6894c064e828ea7129c4ae41752 in term 1.
I20260926 03:37:40.038139 3121 leader_election.cc:302] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c67aa3c91a443793ecd8aa31bebe69, ea5eb6894c064e828ea7129c4ae41752; no voters:
I20260926 03:37:40.038468 3542 raft_consensus.cc:2781] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:40.038990 3542 raft_consensus.cc:686] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 1 LEADER]: Becoming Leader. State: Replica: ea5eb6894c064e828ea7129c4ae41752, State: Running, Role: LEADER
I20260926 03:37:40.039359 3542 consensus_queue.cc:237] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.042793 3031 catalog_manager.cc:5495] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "ea5eb6894c064e828ea7129c4ae41752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:37:40 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:40.109985 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.090010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40903 (local address 127.0.0.1:40960)
0926 03:37:40.090649 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.090671 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.090792 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.096209 (+ 5417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.096214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.097079 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:40.097333 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.097344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.098479 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.098483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.099535 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.099548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.100300 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.100306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.100507 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.101220 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:40.101239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:40.104182 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:40.107401 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:40.107417 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:40.107426 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:40.107843 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:40.108235 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:40.108239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:40.108241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:40.108390 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:40.108932 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:40.108941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:40.109085 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.109236 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.109519 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":341,"threads_started":1}
I20260926 03:37:40.110059 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.090049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.1.221.190:40903)
0926 03:37:40.095882 (+ 5833us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.095887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.095911 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:37:40.095965 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.095969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.096035 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.096146 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.097517 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.098326 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.099668 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.100135 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.100249 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.104400 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.104405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.104409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.104441 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.107203 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.107971 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.107977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.107980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.108102 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.108511 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.108515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.108517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.108788 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.108915 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.109356 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.109664 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":5698,"thread_start_us":5539,"threads_started":1}
I20260926 03:37:40.115597 3031 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40960:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:37:40.116657 3031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:40.123696 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 8818c0f0cc074fda84eacd2fe5b9f60b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305ce1fa7e5d48d69f5986701ae5b88c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:40.123692 3170 tablet_service.cc:1467] Processing CreateTablet for tablet 8818c0f0cc074fda84eacd2fe5b9f60b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305ce1fa7e5d48d69f5986701ae5b88c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:40.124284 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818c0f0cc074fda84eacd2fe5b9f60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:40.124294 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818c0f0cc074fda84eacd2fe5b9f60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:40.124235 3448 tablet_service.cc:1467] Processing CreateTablet for tablet 8818c0f0cc074fda84eacd2fe5b9f60b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305ce1fa7e5d48d69f5986701ae5b88c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:40.124763 3448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818c0f0cc074fda84eacd2fe5b9f60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:40.130968 3539 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: Bootstrap starting.
I20260926 03:37:40.131469 3538 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: Bootstrap starting.
I20260926 03:37:40.132016 3537 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: Bootstrap starting.
I20260926 03:37:40.132750 3539 tablet_bootstrap.cc:654] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:40.133065 3538 tablet_bootstrap.cc:654] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:40.133615 3537 tablet_bootstrap.cc:654] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:40.142102 3539 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: No bootstrap required, opened a new log
I20260926 03:37:40.142309 3539 ts_tablet_manager.cc:1397] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260926 03:37:40.143117 3539 raft_consensus.cc:348] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.143909 3539 raft_consensus.cc:374] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:40.144018 3539 raft_consensus.cc:729] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cfbd44057f455eb9c0a5bb415a236d, State: Initialized, Role: FOLLOWER
I20260926 03:37:40.144306 3539 consensus_queue.cc:260] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.144794 3545 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.145058 3545 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.145192 3538 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: No bootstrap required, opened a new log
I20260926 03:37:40.145205 3537 tablet_bootstrap.cc:492] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: No bootstrap required, opened a new log
I20260926 03:37:40.145364 3538 ts_tablet_manager.cc:1397] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260926 03:37:40.145364 3537 ts_tablet_manager.cc:1397] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260926 03:37:40.146097 3538 raft_consensus.cc:348] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.146088 3537 raft_consensus.cc:348] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.146330 3538 raft_consensus.cc:374] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:40.146440 3538 raft_consensus.cc:729] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c67aa3c91a443793ecd8aa31bebe69, State: Initialized, Role: FOLLOWER
I20260926 03:37:40.146829 3537 raft_consensus.cc:374] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:40.146739 3538 consensus_queue.cc:260] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.146921 3537 raft_consensus.cc:729] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5eb6894c064e828ea7129c4ae41752, State: Initialized, Role: FOLLOWER
I20260926 03:37:40.147178 3537 consensus_queue.cc:260] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.147150 3544 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.147466 3544 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.147648 3542 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.147912 3542 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.148496 3538 ts_tablet_manager.cc:1428] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 03:37:40.149072 3539 ts_tablet_manager.cc:1428] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260926 03:37:40.149315 3537 ts_tablet_manager.cc:1428] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 03:37:40.325516 3545 raft_consensus.cc:482] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:40.325754 3545 raft_consensus.cc:504] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.326481 3545 leader_election.cc:288] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:40.327090 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818c0f0cc074fda84eacd2fe5b9f60b" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5eb6894c064e828ea7129c4ae41752" is_pre_election: true
I20260926 03:37:40.327164 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818c0f0cc074fda84eacd2fe5b9f60b" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69" is_pre_election: true
I20260926 03:37:40.327338 3190 raft_consensus.cc:2454] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 0.
I20260926 03:37:40.327477 3468 raft_consensus.cc:2454] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 0.
I20260926 03:37:40.328207 3261 leader_election.cc:302] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c67aa3c91a443793ecd8aa31bebe69, 96cfbd44057f455eb9c0a5bb415a236d; no voters:
I20260926 03:37:40.328557 3545 raft_consensus.cc:2781] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:40.328671 3545 raft_consensus.cc:482] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:40.328766 3545 raft_consensus.cc:3037] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.330942 3545 raft_consensus.cc:504] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.331558 3545 leader_election.cc:288] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 election: Requested vote from peers ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161), 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673)
I20260926 03:37:40.332113 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818c0f0cc074fda84eacd2fe5b9f60b" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c67aa3c91a443793ecd8aa31bebe69"
I20260926 03:37:40.332208 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818c0f0cc074fda84eacd2fe5b9f60b" candidate_uuid: "96cfbd44057f455eb9c0a5bb415a236d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5eb6894c064e828ea7129c4ae41752"
I20260926 03:37:40.332341 3468 raft_consensus.cc:3037] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.332458 3190 raft_consensus.cc:3037] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:40.334406 3468 raft_consensus.cc:2454] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 1.
I20260926 03:37:40.334896 3261 leader_election.cc:302] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c67aa3c91a443793ecd8aa31bebe69, 96cfbd44057f455eb9c0a5bb415a236d; no voters:
I20260926 03:37:40.335230 3545 raft_consensus.cc:2781] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:40.335408 3545 raft_consensus.cc:686] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [term 1 LEADER]: Becoming Leader. State: Replica: 96cfbd44057f455eb9c0a5bb415a236d, State: Running, Role: LEADER
I20260926 03:37:40.335692 3545 consensus_queue.cc:237] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } }
I20260926 03:37:40.335958 3190 raft_consensus.cc:2454] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cfbd44057f455eb9c0a5bb415a236d in term 1.
I20260926 03:37:40.337530 3558 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: TxnStatusTablet state changed. Reason: New leader 96cfbd44057f455eb9c0a5bb415a236d. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.337814 3558 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:40.338565 3582 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:37:40.338794 3582 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:37:40.339196 3031 catalog_manager.cc:5495] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130). New cstate: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } health_report { overall_health: UNKNOWN } } }
I20260926 03:37:40.388299 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.382647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.1.221.130:35709)
0926 03:37:40.382949 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.382953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.382974 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:37:40.383208 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.383212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.383287 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.383373 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:40.383846 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.384634 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.386493 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.386936 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.387048 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.387638 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.387744 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.387814 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.388027 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.388096 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 03:37:40.388517 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.382578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35709 (local address 127.0.0.1:40356)
0926 03:37:40.382845 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.382866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.383009 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.383449 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.383454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.383468 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:40.383657 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.383667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.384782 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.384787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.386306 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.386323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.387108 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.387114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.387347 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.387462 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.387845 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.387906 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.387995 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.388156 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:37:40.392799 3468 raft_consensus.cc:1261] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Refusing update from remote peer 96cfbd44057f455eb9c0a5bb415a236d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:40.393402 3190 raft_consensus.cc:1261] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Refusing update from remote peer 96cfbd44057f455eb9c0a5bb415a236d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:40.393409 3545 consensus_queue.cc:1035] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:40.393972 3545 consensus_queue.cc:1035] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:40.403707 3544 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: TxnStatusTablet state changed. Reason: New leader 96cfbd44057f455eb9c0a5bb415a236d. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.404037 3544 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.407518 3545 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.407841 3545 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:40.408411 3545 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.408680 3545 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 96cfbd44057f455eb9c0a5bb415a236d: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:40.412068 3542 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: TxnStatusTablet state changed. Reason: New leader 96cfbd44057f455eb9c0a5bb415a236d. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.412385 3542 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.415544 3585 mvcc.cc:205] Tried to move back new op lower bound from 7333453252162437120 to 7333453251938054144. Current Snapshot: MvccSnapshot[applied={T|T < 7333453252162437120}]
I20260926 03:37:40.416316 3544 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.416605 3544 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P 77c67aa3c91a443793ecd8aa31bebe69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.419266 3542 tablet_replica.cc:439] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96cfbd44057f455eb9c0a5bb415a236d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5eb6894c064e828ea7129c4ae41752" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37161 } } peers { permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 } } peers { permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 } } }
I20260926 03:37:40.419624 3542 tablet_replica.cc:442] T 8818c0f0cc074fda84eacd2fe5b9f60b P ea5eb6894c064e828ea7129c4ae41752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:40.428915 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.417940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.1.221.129:37161)
0926 03:37:40.420169 (+ 2229us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.420174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.420799 (+ 625us) server_negotiation.cc:407] Connection header received
0926 03:37:40.421060 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.421066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.421170 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.421360 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:40.422032 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.422854 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.424891 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.426271 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.426495 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.427046 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.428146 (+ 1100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.428297 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.428437 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.428541 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2097,"thread_start_us":1897,"threads_started":1}
I20260926 03:37:40.445813 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.440330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59747 (local address 127.1.221.130:35709)
0926 03:37:40.440521 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.440526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.441058 (+ 532us) server_negotiation.cc:407] Connection header received
0926 03:37:40.441245 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.441248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.441332 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.441408 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:40.441875 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.442632 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.444062 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.444507 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.444621 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.445075 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.445200 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.445274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.445422 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.445502 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:37:40.445889 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.440310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35709 (local address 127.1.221.129:59747)
0926 03:37:40.440939 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.440954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.441106 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.441444 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.441448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.441460 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:40.441727 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.441736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.442766 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.442771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.443912 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.443926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.444689 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.444694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.444836 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.444897 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.445300 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.445311 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.445380 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.445515 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":331,"threads_started":1}
I20260926 03:37:40.464071 3329 raft_consensus.cc:1261] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Refusing update from remote peer ea5eb6894c064e828ea7129c4ae41752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:40.465022 3542 consensus_queue.cc:1035] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cfbd44057f455eb9c0a5bb415a236d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:40.468063 3468 raft_consensus.cc:1261] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Refusing update from remote peer ea5eb6894c064e828ea7129c4ae41752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:40.469105 3542 consensus_queue.cc:1035] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c67aa3c91a443793ecd8aa31bebe69" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:40.531414 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.524286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57915 (local address 127.1.221.129:37161)
0926 03:37:40.524485 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.524491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.524915 (+ 424us) server_negotiation.cc:407] Connection header received
0926 03:37:40.525108 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.525112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.525196 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.525283 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:40.525955 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.527015 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.529391 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.529946 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.530062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.530573 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.530704 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.530789 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.531001 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.531088 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:37:40.532203 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.524159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37161 (local address 127.1.221.130:57915)
0926 03:37:40.524811 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:40.524831 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:40.524972 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:40.525439 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:40.525443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:40.525456 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:40.525809 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.525820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.527128 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.527134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.529249 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:40.529265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.530120 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.530127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:40.530286 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.530380 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:40.530840 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:40.530852 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:37:40.531656 (+ 804us) client_negotiation.cc:240] Negotiation successful
0926 03:37:40.531807 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
I20260926 03:37:40.572389 3033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:40.572652 3033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:40.576916 3033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e6ebf9c33344bcbaa715140648cb82: Sending DeleteTablet for 3 replicas of tablet 01d6f10f70804615be1b3ad80c789c9d
I20260926 03:37:40.577770 3170 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d6f10f70804615be1b3ad80c789c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53058
I20260926 03:37:40.578649 3448 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d6f10f70804615be1b3ad80c789c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:52050
I20260926 03:37:40.580230 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d6f10f70804615be1b3ad80c789c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40338
I20260926 03:37:40.612076 3609 tablet_replica.cc:330] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: stopping tablet replica
I20260926 03:37:40.612455 3609 raft_consensus.cc:2229] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:40.612927 3609 raft_consensus.cc:2258] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:40.615167 3607 tablet_replica.cc:330] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: stopping tablet replica
I20260926 03:37:40.615592 3607 raft_consensus.cc:2229] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:40.616389 3606 tablet_replica.cc:330] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: stopping tablet replica
I20260926 03:37:40.617376 3609 ts_tablet_manager.cc:1905] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:40.617741 3607 raft_consensus.cc:2258] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:40.618208 3606 raft_consensus.cc:2229] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:40.619151 3607 ts_tablet_manager.cc:1905] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:40.619942 3606 raft_consensus.cc:2258] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:40.627825 3606 ts_tablet_manager.cc:1905] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:40.647286 3607 ts_tablet_manager.cc:1918] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:40.647620 3607 log.cc:1196] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/01d6f10f70804615be1b3ad80c789c9d
I20260926 03:37:40.648172 3607 ts_tablet_manager.cc:1939] T 01d6f10f70804615be1b3ad80c789c9d P 77c67aa3c91a443793ecd8aa31bebe69: Deleting consensus metadata
I20260926 03:37:40.649554 3018 catalog_manager.cc:4841] TS 77c67aa3c91a443793ecd8aa31bebe69 (127.1.221.131:36673): tablet 01d6f10f70804615be1b3ad80c789c9d (table test-table [id=15642cc3fcbc4ad58951941a0154cbbc]) successfully deleted
I20260926 03:37:40.650810 3606 ts_tablet_manager.cc:1918] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:40.651044 3609 ts_tablet_manager.cc:1918] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:40.651281 3609 log.cc:1196] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/01d6f10f70804615be1b3ad80c789c9d
I20260926 03:37:40.651974 3609 ts_tablet_manager.cc:1939] T 01d6f10f70804615be1b3ad80c789c9d P 96cfbd44057f455eb9c0a5bb415a236d: Deleting consensus metadata
I20260926 03:37:40.652323 3606 log.cc:1196] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/01d6f10f70804615be1b3ad80c789c9d
I20260926 03:37:40.653718 3017 catalog_manager.cc:4841] TS 96cfbd44057f455eb9c0a5bb415a236d (127.1.221.130:35709): tablet 01d6f10f70804615be1b3ad80c789c9d (table test-table [id=15642cc3fcbc4ad58951941a0154cbbc]) successfully deleted
I20260926 03:37:40.654469 3606 ts_tablet_manager.cc:1939] T 01d6f10f70804615be1b3ad80c789c9d P ea5eb6894c064e828ea7129c4ae41752: Deleting consensus metadata
I20260926 03:37:40.655790 3018 catalog_manager.cc:4841] TS ea5eb6894c064e828ea7129c4ae41752 (127.1.221.129:37161): tablet 01d6f10f70804615be1b3ad80c789c9d (table test-table [id=15642cc3fcbc4ad58951941a0154cbbc]) successfully deleted
Sep 26 03:37:40 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:40.667168 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.592864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.1.221.129:37161)
0926 03:37:40.593104 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.593110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.646436 (+ 53326us) server_negotiation.cc:407] Connection header received
0926 03:37:40.646641 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.646647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.646740 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.646869 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.649482 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.650672 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.652768 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.653439 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.653542 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.658993 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.659015 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.659036 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.659091 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.663494 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.664496 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.664508 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.664512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.664709 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.665169 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.665175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.665179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.665717 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.666050 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.666438 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.666719 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 03:37:40.668077 3235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53078
I20260926 03:37:40.696870 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.671527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.1.221.190:40903)
0926 03:37:40.679880 (+ 8353us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.679887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.679910 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:37:40.680048 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.680055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.680145 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.680270 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.682317 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.683584 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.685417 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.686061 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.686191 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.689583 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.689591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.689597 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.689708 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.693359 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.694230 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.694236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.694239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.694425 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.694905 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.694911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.694915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.695324 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.695642 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.696041 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.696349 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8202,"thread_start_us":8122,"threads_started":1}
W20260926 03:37:40.698285 3033 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40976
I20260926 03:37:40.701208 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3104
I20260926 03:37:40.719578 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.702610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.1.221.129:37161)
0926 03:37:40.702828 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.702834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.703275 (+ 441us) server_negotiation.cc:407] Connection header received
0926 03:37:40.703519 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.703526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.703633 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.703761 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.705209 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.706472 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.708290 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.709349 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.709496 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.712461 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.712468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.712474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.712518 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.716180 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.717126 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.717132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.717136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.717314 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.717784 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.717789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.717793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.718176 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.718449 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.718846 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.719127 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:37:40.720167 3235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658
W20260926 03:37:40.720897 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3104: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:40.721093 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3104
I20260926 03:37:40.737287 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3244
Sep 26 03:37:40 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:37:40.754151 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.738022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.1.221.130:35709)
0926 03:37:40.738198 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.738204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.738276 (+ 72us) server_negotiation.cc:407] Connection header received
0926 03:37:40.738464 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.738469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.738544 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.738635 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.739817 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.740613 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.741827 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.742273 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.742410 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.746841 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.746864 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.746885 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.746938 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.750962 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.751862 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.751873 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.751877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.752061 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.752442 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.752447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.752450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.752970 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.753323 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.753517 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.753779 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:37:40.754944 3374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37794
W20260926 03:37:40.755653 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:40.755820 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3244
I20260926 03:37:40.775362 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3383
Sep 26 03:37:40 dist-test-slave-lqrk krb5kdc[2988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:37:40.790859 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:40.776195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.1.221.131:36673)
0926 03:37:40.776633 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:40.776638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:40.776654 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:37:40.776709 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:40.776713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:40.776772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:40.776863 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:40.778061 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.778758 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.779996 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:40.780469 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:40.780589 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:40.785013 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:40.785030 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:40.785046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:40.785098 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:40.788168 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.789010 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.789020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.789022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.789173 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:40.789526 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:40.789529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:40.789531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:40.789869 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:40.790092 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:40.790338 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 03:37:40.790514 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
W20260926 03:37:40.791595 3513 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57460
W20260926 03:37:40.792179 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:40.792341 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3383
I20260926 03:37:40.805737 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3001
I20260926 03:37:40.806897 3073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:40.921074 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3001
2026-09-26T03:37:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4661 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:41 dist-test-slave-lqrk krb5kdc[3619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:41 dist-test-slave-lqrk krb5kdc[3619](info): set up 2 sockets
Sep 26 03:37:41 dist-test-slave-lqrk krb5kdc[3619](info): commencing operation
krb5kdc: starting...
W20260926 03:37:41.911722 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:42 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:42.091666 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33571
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:42.408448 3632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:42.408771 3632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:42.408897 3632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:42.416137 3632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:42.416263 3632 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:37:42.416323 3632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:42.416407 3632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:42.416473 3632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:42.416559 3632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:42.428155 3632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:42.429742 3632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:42.431003 3632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:42.442689 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:42.442618 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:42.442646 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:42.443317 3632 server_base.cc:1034] running on GCE node
I20260926 03:37:42.444154 3632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:42.445560 3632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:42.446810 3632 hybrid_clock.cc:648] HybridClock initialized: now 1790393862446750 us; error 34 us; skew 500 ppm
I20260926 03:37:42.447144 3632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:42 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:42.450310 3632 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:42.451840 3632 webserver.cc:460] Webserver started at http://127.1.221.190:44443/ using document root <none> and password file <none>
I20260926 03:37:42.452524 3632 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:42.452634 3632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:42.452934 3632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:42.455984 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "539255bb553e43f2a1dc08c7ccfeac64"
format_stamp: "Formatted at 2026-09-26 03:37:42 on dist-test-slave-lqrk"
server_key: "0f6eb04c97aa92b0f027840975064240"
server_key_iv: "17eabfb9bc45445ef3afa013d5cb6dfe"
server_key_version: "encryptionkey@0"
I20260926 03:37:42.456732 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "539255bb553e43f2a1dc08c7ccfeac64"
format_stamp: "Formatted at 2026-09-26 03:37:42 on dist-test-slave-lqrk"
server_key: "0f6eb04c97aa92b0f027840975064240"
server_key_iv: "17eabfb9bc45445ef3afa013d5cb6dfe"
server_key_version: "encryptionkey@0"
I20260926 03:37:42.462005 3632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:42.465562 3647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:42.466889 3632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:42.467049 3632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "539255bb553e43f2a1dc08c7ccfeac64"
format_stamp: "Formatted at 2026-09-26 03:37:42 on dist-test-slave-lqrk"
server_key: "0f6eb04c97aa92b0f027840975064240"
server_key_iv: "17eabfb9bc45445ef3afa013d5cb6dfe"
server_key_version: "encryptionkey@0"
I20260926 03:37:42.467247 3632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:42.501842 3632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:42.506095 3632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:42.506354 3632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:42.525998 3632 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33571
I20260926 03:37:42.526036 3708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33571 every 8 connection(s)
I20260926 03:37:42.527338 3632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:42.530089 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3632
I20260926 03:37:42.530305 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:42.530944 1910 external_mini_cluster.cc:1428] Setting key 25449a66bd80b89ada0dae235f2c686a
I20260926 03:37:42.531312 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:42.545351 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Bootstrap starting.
Sep 26 03:37:42 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:42.548609 3709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:42.549639 3709 log.cc:824] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:42.552388 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: No bootstrap required, opened a new log
I20260926 03:37:42.554050 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:42.532916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.1.221.190:33571)
0926 03:37:42.533684 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:42.533696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:42.533739 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:37:42.536115 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:42.536143 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:42.536557 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:42.536935 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:42.538706 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:42.540023 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:42.541764 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:42.542471 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:42.542612 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:42.547090 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:42.547100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:42.547117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:42.547160 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:42.550066 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:42.550756 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:42.550763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:42.550770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:42.550934 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:42.551295 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:42.551299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:42.551301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:42.551882 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:42.552145 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:42.552607 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 03:37:42.553019 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":359,"threads_started":1}
I20260926 03:37:42.560740 3709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } }
I20260926 03:37:42.561115 3709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:42.561210 3709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539255bb553e43f2a1dc08c7ccfeac64, State: Initialized, Role: FOLLOWER
I20260926 03:37:42.561789 3709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } }
I20260926 03:37:42.562005 3709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:42.562126 3709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:42.562281 3709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:42.564972 3709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } }
I20260926 03:37:42.565493 3709 leader_election.cc:302] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 539255bb553e43f2a1dc08c7ccfeac64; no voters:
I20260926 03:37:42.566054 3709 leader_election.cc:288] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:42.566181 3714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:42.566898 3714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [term 1 LEADER]: Becoming Leader. State: Replica: 539255bb553e43f2a1dc08c7ccfeac64, State: Running, Role: LEADER
I20260926 03:37:42.567454 3714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } }
I20260926 03:37:42.568136 3709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:42.570032 3715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "539255bb553e43f2a1dc08c7ccfeac64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } } }
I20260926 03:37:42.570395 3715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:42.571153 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 539255bb553e43f2a1dc08c7ccfeac64. Latest consensus state: current_term: 1 leader_uuid: "539255bb553e43f2a1dc08c7ccfeac64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539255bb553e43f2a1dc08c7ccfeac64" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33571 } } }
I20260926 03:37:42.571430 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:42.572235 3719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:42.576905 3719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:42.586480 3719 catalog_manager.cc:1347] Generated new cluster ID: 87a7b937b2fe4483bd6468d1a3251b31
I20260926 03:37:42.586616 3719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:42.619402 3719 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:42.620781 3719 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:42.627045 3719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Generated new TSK 0
I20260926 03:37:42.627962 3719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:42.684114 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33571
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:43.005578 3735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:43.005879 3735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:43.006007 3735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:43.013458 3735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:43.013577 3735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:43.013679 3735 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:37:43.013763 3735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:43.013866 3735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:43.023730 3735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:43.025071 3735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:43.026373 3735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:43.038954 3740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:43.038987 3743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:43.038925 3741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:43.039812 3735 server_base.cc:1034] running on GCE node
I20260926 03:37:43.040380 3735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:43.041288 3735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:43.042510 3735 hybrid_clock.cc:648] HybridClock initialized: now 1790393863042452 us; error 50 us; skew 500 ppm
I20260926 03:37:43.042843 3735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:43 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:43.045885 3735 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:43.047333 3735 webserver.cc:460] Webserver started at http://127.1.221.129:37197/ using document root <none> and password file <none>
I20260926 03:37:43.048013 3735 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:43.048125 3735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:43.048415 3735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:43.051208 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "04e3dc77103a65c84f59c4d4eee03c3b"
server_key_iv: "37430548a6a09fe61fe590a2f13439c1"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.051991 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "04e3dc77103a65c84f59c4d4eee03c3b"
server_key_iv: "37430548a6a09fe61fe590a2f13439c1"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.057163 3735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:37:43.060688 3750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:43.062124 3735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:43.062278 3735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "04e3dc77103a65c84f59c4d4eee03c3b"
server_key_iv: "37430548a6a09fe61fe590a2f13439c1"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.062454 3735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:43.075870 3735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:43.080266 3735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:43.080567 3735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:43.083333 3735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:43.083482 3735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:37:43.083597 3735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:43.083711 3735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:37:43 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:43.105000 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.083848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47633 (local address 127.1.221.190:33571)
0926 03:37:43.084531 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:43.084535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:43.085525 (+ 990us) server_negotiation.cc:407] Connection header received
0926 03:37:43.087539 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:43.087544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:43.087636 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:43.087735 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:43.089899 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.091028 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.093477 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.094028 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.094148 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.098911 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:43.098917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:43.098920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:43.098954 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:43.101687 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.102425 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.102431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.102435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.102558 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.102995 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.103000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.103003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.103300 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:43.103463 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:43.104322 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 03:37:43.104571 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":503,"threads_started":1}
I20260926 03:37:43.106004 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.084564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.129:47633)
0926 03:37:43.085356 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:43.085401 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:43.087215 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:43.088014 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:43.088023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:43.088777 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:43.089629 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.089649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.091221 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.091227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.093332 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.093348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.094482 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.094487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.094777 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.095640 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:43.095673 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:43.098664 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:43.101833 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.101844 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.101862 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.102312 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.102666 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.102672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.102676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.102870 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.103459 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:43.103468 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:43.103860 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 03:37:43.104166 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 03:37:43.104773 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":342,"threads_started":1}
I20260926 03:37:43.124373 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.111820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33767 (local address 127.1.221.190:33571)
0926 03:37:43.112024 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:43.112030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:43.112397 (+ 367us) server_negotiation.cc:407] Connection header received
0926 03:37:43.112576 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:43.112582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:43.112665 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:43.112776 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:43.113643 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.114788 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.116311 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.116756 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.116867 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.118622 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:43.118628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:43.118632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:43.118662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:43.121598 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.122584 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.122588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.122590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.122723 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.123114 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.123117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.123119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.123427 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:43.123538 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:43.123976 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 03:37:43.124142 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:37:43.124729 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.111579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.129:33767)
0926 03:37:43.112301 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:43.112320 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:43.112460 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:43.112794 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:43.112797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:43.113240 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:43.113517 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.113525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.114926 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.114930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.116150 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.116165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.116936 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.116943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.117113 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.117677 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:43.117693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:43.118441 (+ 748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:43.121805 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.121811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.121815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.122434 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.122833 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.122839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.122843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.123011 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.123526 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:43.123531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:43.123745 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 03:37:43.123899 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 03:37:43.124257 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":442,"threads_started":1}
I20260926 03:37:43.137738 3735 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:45337
I20260926 03:37:43.137776 3870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:45337 every 8 connection(s)
I20260926 03:37:43.138980 3735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:43.141037 3871 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33571
I20260926 03:37:43.141238 3871 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:43.141820 3871 heartbeater.cc:511] Master 127.1.221.190:33571 requested a full tablet report, sending...
I20260926 03:37:43.143615 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3735
I20260926 03:37:43.143815 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:43.143893 3664 ts_manager.cc:194] Registered new tserver with Master: e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337)
I20260926 03:37:43.144593 1910 external_mini_cluster.cc:1428] Setting key 2ec9f65d3a104fe26573eefec4ca1611
I20260926 03:37:43.145741 3664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:47633
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:43.182699 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33571
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:43.496446 3875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:43.496740 3875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:43.496892 3875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:43.504038 3875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:43.504170 3875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:43.504299 3875 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:37:43.504364 3875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:43.504470 3875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:43.514418 3875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:43.515801 3875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:43.517134 3875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:43.529536 3881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:43.529685 3883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:43.530006 3880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:43.530730 3875 server_base.cc:1034] running on GCE node
I20260926 03:37:43.531440 3875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:43.532536 3875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:43.533769 3875 hybrid_clock.cc:648] HybridClock initialized: now 1790393863533708 us; error 51 us; skew 500 ppm
I20260926 03:37:43.534106 3875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:43 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:43.537720 3875 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:43.539417 3875 webserver.cc:460] Webserver started at http://127.1.221.130:45115/ using document root <none> and password file <none>
I20260926 03:37:43.540239 3875 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:43.540388 3875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:43.540720 3875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:43.544512 3875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "dd84cbc20fda4c1da0fc387b0231b711"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "8d7c028ee2aabe33da7e68f07d2854ea"
server_key_iv: "7ccf0c51c1613db5c6f9c5856a62b1f4"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.545286 3875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "dd84cbc20fda4c1da0fc387b0231b711"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "8d7c028ee2aabe33da7e68f07d2854ea"
server_key_iv: "7ccf0c51c1613db5c6f9c5856a62b1f4"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.550172 3875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:37:43.553561 3890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:43.554890 3875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:43.555038 3875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "dd84cbc20fda4c1da0fc387b0231b711"
format_stamp: "Formatted at 2026-09-26 03:37:43 on dist-test-slave-lqrk"
server_key: "8d7c028ee2aabe33da7e68f07d2854ea"
server_key_iv: "7ccf0c51c1613db5c6f9c5856a62b1f4"
server_key_version: "encryptionkey@0"
I20260926 03:37:43.555229 3875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:43.604576 3875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:43.608913 3875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:43.609185 3875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:43.611982 3875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:43.612102 3875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:43.612217 3875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:43.612308 3875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:37:43 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:43.635237 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.612508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:44987 (local address 127.1.221.190:33571)
0926 03:37:43.612731 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:43.612737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:43.614209 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 03:37:43.616389 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:43.616395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:43.616490 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:43.616608 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:43.618832 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.619763 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.622122 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.622711 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.622835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.628160 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:43.628166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:43.628169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:43.628204 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:43.631728 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.632691 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.632698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.632702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.632865 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.633398 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.633404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.633408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.633688 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:43.633833 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:43.634673 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 03:37:43.634899 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:37:43.636673 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.613244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.130:44987)
0926 03:37:43.614032 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:43.614071 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:43.616061 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:43.616995 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:43.617003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:43.617762 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:43.618624 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.618654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.619964 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.619968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.621962 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.621974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.623063 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.623073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.623468 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.624366 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:43.624401 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:43.627852 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:43.631884 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.631894 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.631917 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.632543 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.632999 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.633004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.633008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.633267 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.633833 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:43.633841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:43.634254 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 03:37:43.634558 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 03:37:43.635088 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":348,"threads_started":1}
I20260926 03:37:43.655342 3875 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:41521
I20260926 03:37:43.655421 4009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:41521 every 8 connection(s)
I20260926 03:37:43.656873 3875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:43.657799 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.643318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55787 (local address 127.1.221.190:33571)
0926 03:37:43.643550 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:43.643557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:43.644383 (+ 826us) server_negotiation.cc:407] Connection header received
0926 03:37:43.644505 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:43.644510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:43.644599 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:43.644707 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:43.645816 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.646929 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.649289 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.649709 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.649809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.652160 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:43.652167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:43.652173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:43.652236 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:43.654981 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.655850 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.655854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.655856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.655982 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:43.656355 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:43.656358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:43.656360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:43.656662 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:43.656773 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:43.657268 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 03:37:43.657480 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:37:43.658010 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:43.643312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.130:55787)
0926 03:37:43.644272 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:43.644289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:43.644405 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:43.644773 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:43.644777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:43.645357 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:43.645645 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.645657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.647136 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.647141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.649124 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:43.649143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:43.649916 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:43.649923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:43.650116 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:43.650757 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:43.650782 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:43.651950 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:43.655102 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.655108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.655113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.655719 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.656101 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:43.656105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:43.656108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:43.656258 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:43.656787 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:43.656792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:43.656999 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 03:37:43.657157 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 03:37:43.657521 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":575,"threads_started":1}
I20260926 03:37:43.659446 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 3875
I20260926 03:37:43.659605 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:43.660204 1910 external_mini_cluster.cc:1428] Setting key a75628a4c8809419f05442da57027ec0
I20260926 03:37:43.660950 4010 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33571
I20260926 03:37:43.661196 4010 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:43.661995 4010 heartbeater.cc:511] Master 127.1.221.190:33571 requested a full tablet report, sending...
I20260926 03:37:43.663590 3664 ts_manager.cc:194] Registered new tserver with Master: dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:43.664347 3664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:44987
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:43.699747 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33571
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:44.017153 4014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:44.017433 4014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:44.017567 4014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:44.024914 4014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:44.025031 4014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:44.025159 4014 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:37:44.025218 4014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:37:44.025321 4014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:44.035441 4014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:44.036813 4014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:44.038101 4014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:44.051085 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:44.051085 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:44.051134 4022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:44.051834 4014 server_base.cc:1034] running on GCE node
I20260926 03:37:44.052446 4014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:44.053423 4014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:44.054622 4014 hybrid_clock.cc:648] HybridClock initialized: now 1790393864054561 us; error 55 us; skew 500 ppm
I20260926 03:37:44.054941 4014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:44 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393864, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:44.063593 4014 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:44.065217 4014 webserver.cc:460] Webserver started at http://127.1.221.131:44989/ using document root <none> and password file <none>
I20260926 03:37:44.065908 4014 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:44.066035 4014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:44.066365 4014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:44.069589 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "35294c46c02e422d979a2796c96a5e6b"
format_stamp: "Formatted at 2026-09-26 03:37:44 on dist-test-slave-lqrk"
server_key: "9d17a2869315753f97c8281f7848f9e7"
server_key_iv: "65d26a6d8574c855f246438898394f38"
server_key_version: "encryptionkey@0"
I20260926 03:37:44.070331 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "35294c46c02e422d979a2796c96a5e6b"
format_stamp: "Formatted at 2026-09-26 03:37:44 on dist-test-slave-lqrk"
server_key: "9d17a2869315753f97c8281f7848f9e7"
server_key_iv: "65d26a6d8574c855f246438898394f38"
server_key_version: "encryptionkey@0"
I20260926 03:37:44.075647 4014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:37:44.079291 4029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:44.080749 4014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:44.080901 4014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "35294c46c02e422d979a2796c96a5e6b"
format_stamp: "Formatted at 2026-09-26 03:37:44 on dist-test-slave-lqrk"
server_key: "9d17a2869315753f97c8281f7848f9e7"
server_key_iv: "65d26a6d8574c855f246438898394f38"
server_key_version: "encryptionkey@0"
I20260926 03:37:44.081086 4014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:44.098945 4014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:44.103991 4014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:44.104249 4014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:44.106832 4014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:44.106948 4014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:44.107064 4014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:44.107146 4014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:37:44 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393864, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:44.130379 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.107483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:33881 (local address 127.1.221.190:33571)
0926 03:37:44.107686 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.107691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.109210 (+ 1519us) server_negotiation.cc:407] Connection header received
0926 03:37:44.111181 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.111188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.111278 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.111420 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:44.113439 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.114677 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.117134 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.117756 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.117883 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.122743 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:44.122750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:44.122755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:44.122794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:44.126172 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.127554 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.127560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.127564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.127737 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.128282 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.128287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.128291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.128640 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:44.128792 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.129768 (+ 976us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.130001 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:37:44.131953 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.108197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.131:33881)
0926 03:37:44.109009 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.109055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.110886 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.111662 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.111673 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.112377 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:44.113247 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.113277 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.114863 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.114869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.116948 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.116963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.118142 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.118149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.118340 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.119119 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:44.119140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:44.122447 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:44.126676 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.126698 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.126718 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.127340 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.127851 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.127857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.127860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.128164 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.128860 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:44.128873 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:44.129367 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.129616 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.130227 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":316,"threads_started":1}
I20260926 03:37:44.147229 4014 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:42807
I20260926 03:37:44.147279 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:42807 every 8 connection(s)
I20260926 03:37:44.148530 3871 heartbeater.cc:503] Master 127.1.221.190:33571 was elected leader, sending a full tablet report...
I20260926 03:37:44.148610 4014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:44.151345 4149 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33571
I20260926 03:37:44.151589 4149 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:44.152235 4149 heartbeater.cc:511] Master 127.1.221.190:33571 requested a full tablet report, sending...
I20260926 03:37:44.153674 3664 ts_manager.cc:194] Registered new tserver with Master: 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
I20260926 03:37:44.154702 3664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:33881
I20260926 03:37:44.157078 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4014
I20260926 03:37:44.157255 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:44.157845 1910 external_mini_cluster.cc:1428] Setting key b73d88acb93f5f15bde202355262d3cd
I20260926 03:37:44.161245 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:37:44.162159 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.143056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60899 (local address 127.1.221.190:33571)
0926 03:37:44.143264 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.143272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.149451 (+ 6179us) server_negotiation.cc:407] Connection header received
0926 03:37:44.149644 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.149649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.149733 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.149835 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:44.151262 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.152123 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.153602 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.154073 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.154176 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.156067 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:44.156072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:44.156076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:44.156105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:44.159288 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.160136 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.160142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.160146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.160346 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.160713 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.160718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.160721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.161025 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:44.161175 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.161594 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.161797 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:37:44.162209 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.142634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.1.221.131:60899)
0926 03:37:44.149316 (+ 6682us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.149345 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.149509 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.150007 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.150012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.150626 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:44.151093 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.151108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.152271 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.152275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.153442 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.153456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.154256 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.154263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.154432 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.155058 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:44.155081 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:44.155888 (+ 807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:44.159474 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.159497 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.159501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.160018 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.160457 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.160462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.160465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.160586 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.161156 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:44.161162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:44.161332 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.161468 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.161788 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6504,"thread_start_us":6330,"threads_started":1}
Sep 26 03:37:44 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:37:44 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:44.206787 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.192362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.1.221.190:33571)
0926 03:37:44.192602 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.192618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.192731 (+ 113us) server_negotiation.cc:407] Connection header received
0926 03:37:44.192872 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.192875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.192940 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.193023 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:44.194439 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.195208 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.196413 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.196839 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.196933 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.201645 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:44.201651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:44.201655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:44.201690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:44.204202 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.204949 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.204953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.204956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.205086 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.205440 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.205443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.205445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.205744 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:44.205923 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.206210 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.206393 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 03:37:44.210744 3664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:44.217466 3664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:44.240444 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.229715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.1.221.130:41521)
0926 03:37:44.230856 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.230867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.233256 (+ 2389us) server_negotiation.cc:407] Connection header received
0926 03:37:44.233362 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.233371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.233625 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.233790 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.234448 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.235926 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.237685 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.239089 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.239215 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.239462 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.239887 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.240066 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":793,"threads_started":1}
I20260926 03:37:44.240463 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.229543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41521 (local address 127.0.0.1:41806)
0926 03:37:44.232795 (+ 3252us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.232849 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.233049 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.233844 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.233852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.233879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.234282 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.234294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.236068 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.236071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.237529 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.237540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.239325 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.239332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.239625 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.239684 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.239824 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.240030 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2948,"mutex_wait_us":27,"thread_start_us":4541,"threads_started":1}
I20260926 03:37:44.243616 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.230035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42807 (local address 127.0.0.1:57982)
0926 03:37:44.234354 (+ 4319us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.234370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.234556 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.235435 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.235440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.235464 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.235844 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.235855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.237959 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.237964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.240086 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.240099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.242775 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.242782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.242967 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.242999 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.243077 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.243206 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4197,"mutex_wait_us":59,"thread_start_us":2318,"threads_started":1}
I20260926 03:37:44.243709 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.230698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.1.221.131:42807)
0926 03:37:44.233624 (+ 2926us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.233633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.234456 (+ 823us) server_negotiation.cc:407] Connection header received
0926 03:37:44.234724 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.234732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.234959 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.235145 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.236082 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.237808 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.240295 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.242285 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.242406 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.242628 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.243185 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.243350 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2740,"thread_start_us":2549,"threads_started":1}
I20260926 03:37:44.244405 3940 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 (DEFAULT_TABLE table=test-table [id=38b3abbc5b8d4bc29147a222ea52b273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.245714 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.235313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45337 (local address 127.0.0.1:60974)
0926 03:37:44.238047 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.238064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.238185 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.238936 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.238940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.238968 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.239473 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.239486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.241305 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.241311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.243289 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.243302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.244943 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.244949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.245116 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.245149 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.245233 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.245364 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2613,"thread_start_us":2470,"threads_started":1}
I20260926 03:37:44.245723 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.234747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.1.221.129:45337)
0926 03:37:44.235284 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.235293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.238303 (+ 3010us) server_negotiation.cc:407] Connection header received
0926 03:37:44.238398 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.238405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.238653 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.238831 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.239668 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.241163 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.243476 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.244737 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.244863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.245061 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.245294 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.245455 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":286,"threads_started":1}
I20260926 03:37:44.245824 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f8bbad9744d51ae43303f54b9f801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.249099 3801 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 (DEFAULT_TABLE table=test-table [id=38b3abbc5b8d4bc29147a222ea52b273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.249339 4079 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 (DEFAULT_TABLE table=test-table [id=38b3abbc5b8d4bc29147a222ea52b273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.250346 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f8bbad9744d51ae43303f54b9f801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.250741 4079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f8bbad9744d51ae43303f54b9f801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.262100 4168 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Bootstrap starting.
I20260926 03:37:44.265671 4170 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Bootstrap starting.
I20260926 03:37:44.267355 4169 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Bootstrap starting.
I20260926 03:37:44.267408 4168 tablet_bootstrap.cc:654] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.268857 4168 log.cc:824] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:44.269168 4170 tablet_bootstrap.cc:654] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.270291 4169 tablet_bootstrap.cc:654] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.270427 4170 log.cc:824] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:44.271601 4169 log.cc:824] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:44.274358 4169 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: No bootstrap required, opened a new log
I20260926 03:37:44.274617 4168 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: No bootstrap required, opened a new log
I20260926 03:37:44.274612 4170 tablet_bootstrap.cc:492] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: No bootstrap required, opened a new log
I20260926 03:37:44.274621 4169 ts_tablet_manager.cc:1397] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 03:37:44.274883 4168 ts_tablet_manager.cc:1397] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260926 03:37:44.274883 4170 ts_tablet_manager.cc:1397] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 03:37:44.282152 4169 raft_consensus.cc:348] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.282527 4169 raft_consensus.cc:374] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.282614 4169 raft_consensus.cc:729] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd84cbc20fda4c1da0fc387b0231b711, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.283201 4169 consensus_queue.cc:260] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.284261 4010 heartbeater.cc:503] Master 127.1.221.190:33571 was elected leader, sending a full tablet report...
I20260926 03:37:44.285023 4169 ts_tablet_manager.cc:1428] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Time spent starting tablet: real 0.010s user 0.010s sys 0.002s
I20260926 03:37:44.285267 4168 raft_consensus.cc:348] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.285271 4170 raft_consensus.cc:348] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.285699 4170 raft_consensus.cc:374] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.285699 4168 raft_consensus.cc:374] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.285797 4168 raft_consensus.cc:729] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.285797 4170 raft_consensus.cc:729] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.286391 4170 consensus_queue.cc:260] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.286391 4168 consensus_queue.cc:260] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.288151 4170 ts_tablet_manager.cc:1428] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 03:37:44.288477 4168 ts_tablet_manager.cc:1428] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260926 03:37:44.288964 4149 heartbeater.cc:503] Master 127.1.221.190:33571 was elected leader, sending a full tablet report...
W20260926 03:37:44.391324 3872 tablet.cc:2369] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:37:44.403565 4150 tablet.cc:2369] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:37:44.410737 4011 tablet.cc:2369] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:44.440904 4176 raft_consensus.cc:482] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:44.441203 4176 raft_consensus.cc:504] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.442989 4176 leader_election.cc:288] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521), 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
I20260926 03:37:44.451929 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.443436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33995 (local address 127.1.221.130:41521)
0926 03:37:44.443699 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.443707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.444574 (+ 867us) server_negotiation.cc:407] Connection header received
0926 03:37:44.444919 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.444924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.444995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.445081 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.445663 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.446727 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.449621 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.450803 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.450970 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.451042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.451533 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.451670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:37:44.451923 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.443779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59069 (local address 127.1.221.131:42807)
0926 03:37:44.444029 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.444035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.444589 (+ 554us) server_negotiation.cc:407] Connection header received
0926 03:37:44.444918 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.444924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.445004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.445074 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.445848 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.446888 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.449636 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.450810 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.450973 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.451042 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.451533 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.451687 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:37:44.452170 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.443315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41521 (local address 127.1.221.129:33995)
0926 03:37:44.444471 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.444489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.444718 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.445107 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.445111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.445133 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.445511 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.445520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.446876 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.446880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.449442 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.449453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.451067 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.451077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.451335 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.451406 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.451504 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.451716 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":165,"spinlock_wait_cycles":30592,"thread_start_us":875,"threads_started":1}
I20260926 03:37:44.452368 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.443277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42807 (local address 127.1.221.129:59069)
0926 03:37:44.444471 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.444492 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.444736 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.445260 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.445267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.445305 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.445692 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.445701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.447040 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.447045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.449429 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.449447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.451092 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.451098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.451328 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.451406 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.451504 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.451716 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":74,"spinlock_wait_cycles":30336,"thread_start_us":1142,"threads_started":1}
I20260926 03:37:44.453212 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f8bbad9744d51ae43303f54b9f801" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
I20260926 03:37:44.453222 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f8bbad9744d51ae43303f54b9f801" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35294c46c02e422d979a2796c96a5e6b" is_pre_election: true
I20260926 03:37:44.453668 4099 raft_consensus.cc:2454] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 0.
I20260926 03:37:44.453667 3960 raft_consensus.cc:2454] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 0.
I20260926 03:37:44.454273 3751 leader_election.cc:302] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, e3decc86327b4ef994f72dcc7d3adc4a; no voters:
I20260926 03:37:44.454604 4176 raft_consensus.cc:2781] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:44.454738 4176 raft_consensus.cc:482] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:44.454838 4176 raft_consensus.cc:3037] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.456969 4176 raft_consensus.cc:504] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.457552 4176 leader_election.cc:288] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 election: Requested vote from peers dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521), 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
I20260926 03:37:44.458026 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f8bbad9744d51ae43303f54b9f801" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711"
I20260926 03:37:44.458050 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f8bbad9744d51ae43303f54b9f801" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35294c46c02e422d979a2796c96a5e6b"
I20260926 03:37:44.458195 3960 raft_consensus.cc:3037] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.458232 4099 raft_consensus.cc:3037] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.460525 4099 raft_consensus.cc:2454] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 1.
I20260926 03:37:44.460603 3960 raft_consensus.cc:2454] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 1.
I20260926 03:37:44.460909 3751 leader_election.cc:302] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, e3decc86327b4ef994f72dcc7d3adc4a; no voters:
I20260926 03:37:44.461205 4176 raft_consensus.cc:2781] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:44.461653 4176 raft_consensus.cc:686] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 LEADER]: Becoming Leader. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Running, Role: LEADER
I20260926 03:37:44.462093 4176 consensus_queue.cc:237] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.466997 3662 catalog_manager.cc:5495] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129). New cstate: current_term: 1 leader_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:37:44 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:44.518875 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.506007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.1.221.190:33571)
0926 03:37:44.506184 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.506188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.506708 (+ 520us) server_negotiation.cc:407] Connection header received
0926 03:37:44.506881 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.506886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.506946 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.507028 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:44.508356 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.509141 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.510356 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.510767 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.510871 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.514203 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:44.514209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:44.514213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:44.514243 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:44.516538 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.517192 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.517196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.517198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.517316 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:44.517613 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:44.517616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:44.517618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:44.517881 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:44.517983 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.518366 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.518610 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":57}
I20260926 03:37:44.518893 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.505969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33571 (local address 127.0.0.1:35342)
0926 03:37:44.506598 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.506615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.506770 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.507083 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.507087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.507886 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:44.508187 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.508198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.509271 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.509275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.510218 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.510229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.510949 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.510954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.511085 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.511836 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:44.511853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:44.514001 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:44.516659 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.516670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.516675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.517080 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.517404 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:44.517407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:44.517409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:44.517532 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:44.517981 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:44.517986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:44.518151 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.518285 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.518488 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":350,"threads_started":1}
I20260926 03:37:44.523846 3662 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:35342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:37:44.524870 3662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:44.531618 3801 tablet_service.cc:1467] Processing CreateTablet for tablet af9999d0ac194d54a467898818249036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bb4a660cc146cfad7f090b2388713c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:44.531661 3940 tablet_service.cc:1467] Processing CreateTablet for tablet af9999d0ac194d54a467898818249036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bb4a660cc146cfad7f090b2388713c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:44.532414 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9999d0ac194d54a467898818249036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.532510 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9999d0ac194d54a467898818249036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.535074 4079 tablet_service.cc:1467] Processing CreateTablet for tablet af9999d0ac194d54a467898818249036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67bb4a660cc146cfad7f090b2388713c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:37:44.536068 4079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9999d0ac194d54a467898818249036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.538861 4168 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: Bootstrap starting.
I20260926 03:37:44.539443 4169 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: Bootstrap starting.
I20260926 03:37:44.540475 4168 tablet_bootstrap.cc:654] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.540784 4169 tablet_bootstrap.cc:654] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.542685 4168 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: No bootstrap required, opened a new log
I20260926 03:37:44.542820 4168 ts_tablet_manager.cc:1397] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:37:44.542922 4170 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: Bootstrap starting.
I20260926 03:37:44.543578 4168 raft_consensus.cc:348] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.543817 4168 raft_consensus.cc:374] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.543905 4168 raft_consensus.cc:729] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.544137 4168 consensus_queue.cc:260] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.544520 4176 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.544749 4170 tablet_bootstrap.cc:654] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.544780 4176 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.545679 4168 ts_tablet_manager.cc:1428] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:44.547164 4169 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: No bootstrap required, opened a new log
I20260926 03:37:44.547307 4169 ts_tablet_manager.cc:1397] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 03:37:44.548029 4169 raft_consensus.cc:348] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.548254 4169 raft_consensus.cc:374] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.548333 4169 raft_consensus.cc:729] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd84cbc20fda4c1da0fc387b0231b711, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.548599 4169 consensus_queue.cc:260] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.549146 4174 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.549486 4174 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.550416 4170 tablet_bootstrap.cc:492] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: No bootstrap required, opened a new log
I20260926 03:37:44.550464 4169 ts_tablet_manager.cc:1428] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:37:44.550654 4170 ts_tablet_manager.cc:1397] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:37:44.551440 4170 raft_consensus.cc:348] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.551688 4170 raft_consensus.cc:374] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.551786 4170 raft_consensus.cc:729] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.552006 4170 consensus_queue.cc:260] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.552577 4170 ts_tablet_manager.cc:1428] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:37:44.552487 4175 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.552826 4175 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.596596 4175 raft_consensus.cc:482] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:44.596786 4175 raft_consensus.cc:504] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.598217 4175 leader_election.cc:288] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:44.603992 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.598547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45337 (local address 127.1.221.131:47607)
0926 03:37:44.598722 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.598742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.598858 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.599265 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.599271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.599300 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.599581 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.599593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.600656 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.600660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.601987 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.601999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.603193 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.603199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.603421 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.603447 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.603542 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.603685 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:37:44.603988 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.598709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:47607 (local address 127.1.221.129:45337)
0926 03:37:44.598922 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.598929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.598951 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:37:44.599030 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.599044 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.599128 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.599250 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.599751 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.600497 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.602187 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.603030 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.603192 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.603277 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.603596 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.603728 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:37:44.604640 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.598932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41521 (local address 127.1.221.131:53069)
0926 03:37:44.599741 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.599755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.599867 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.600152 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.600156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.600188 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:37:44.600492 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.600500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.601502 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.601506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.602775 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.602786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.603935 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.603941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.604094 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.604125 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.604261 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.604384 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":1,"thread_start_us":543,"threads_started":1}
I20260926 03:37:44.604764 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.599081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53069 (local address 127.1.221.130:41521)
0926 03:37:44.599306 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.599313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.599843 (+ 530us) server_negotiation.cc:407] Connection header received
0926 03:37:44.599986 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.599991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.600052 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.600122 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:37:44.600632 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.601368 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.602919 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.603730 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.603870 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.603937 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.604306 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.604449 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:37:44.605157 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9999d0ac194d54a467898818249036" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" is_pre_election: true
I20260926 03:37:44.605407 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9999d0ac194d54a467898818249036" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
I20260926 03:37:44.605630 3960 raft_consensus.cc:2454] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:44.605587 3826 raft_consensus.cc:2454] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:44.606050 4033 leader_election.cc:302] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711; no voters:
I20260926 03:37:44.606431 4175 raft_consensus.cc:2781] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:44.606603 4175 raft_consensus.cc:482] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:44.606706 4175 raft_consensus.cc:3037] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.609337 4175 raft_consensus.cc:504] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.609988 4175 leader_election.cc:288] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Requested vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:44.610436 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9999d0ac194d54a467898818249036" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
I20260926 03:37:44.610496 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9999d0ac194d54a467898818249036" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711"
I20260926 03:37:44.610648 3826 raft_consensus.cc:3037] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.610720 3960 raft_consensus.cc:3037] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:44.612779 3960 raft_consensus.cc:2454] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:44.612793 3826 raft_consensus.cc:2454] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:44.613173 4033 leader_election.cc:302] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711; no voters:
I20260926 03:37:44.613538 4175 raft_consensus.cc:2781] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:44.613970 4175 raft_consensus.cc:686] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [term 1 LEADER]: Becoming Leader. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Running, Role: LEADER
I20260926 03:37:44.614434 4175 consensus_queue.cc:237] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.617527 4191 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: TxnStatusTablet state changed. Reason: New leader 35294c46c02e422d979a2796c96a5e6b. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.617839 4191 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:44.618505 4193 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:37:44.618764 4193 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:37:44.619359 3662 catalog_manager.cc:5495] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 35294c46c02e422d979a2796c96a5e6b (127.1.221.131). New cstate: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } health_report { overall_health: HEALTHY } } }
I20260926 03:37:44.638332 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.631928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42807 (local address 127.0.0.1:57990)
0926 03:37:44.632166 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.632190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.632325 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.632832 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.632836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.632847 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:44.633019 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.633027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.634532 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.634536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.635694 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.635707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.636679 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.636684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.636840 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.636898 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.637896 (+ 998us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.637914 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.637979 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.638093 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 03:37:44.638399 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.632092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.1.221.131:42807)
0926 03:37:44.632305 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.632311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.632338 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:37:44.632492 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.632497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.632621 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.632796 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:44.633157 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.634396 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.635828 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.636507 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.636632 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.637079 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.637783 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.637925 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.638004 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.638086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:37:44.645965 3960 raft_consensus.cc:1261] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:44.645998 3826 raft_consensus.cc:1261] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:44.646847 4191 consensus_queue.cc:1035] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:44.647147 4175 consensus_queue.cc:1035] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:44.654232 4174 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: TxnStatusTablet state changed. Reason: New leader 35294c46c02e422d979a2796c96a5e6b. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.660683 4174 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.662495 4196 mvcc.cc:205] Tried to move back new op lower bound from 7333453269580603392 to 7333453269464461312. Current Snapshot: MvccSnapshot[applied={T|T < 7333453269580603392}]
I20260926 03:37:44.663681 4197 mvcc.cc:205] Tried to move back new op lower bound from 7333453269580603392 to 7333453269464461312. Current Snapshot: MvccSnapshot[applied={T|T < 7333453269580603392}]
I20260926 03:37:44.667894 4174 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.668126 4174 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P dd84cbc20fda4c1da0fc387b0231b711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.670439 4176 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: TxnStatusTablet state changed. Reason: New leader 35294c46c02e422d979a2796c96a5e6b. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.670766 4176 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.673492 4198 mvcc.cc:205] Tried to move back new op lower bound from 7333453269580603392 to 7333453269464461312. Current Snapshot: MvccSnapshot[applied={T|T < 7333453269580603392}]
I20260926 03:37:44.682667 4191 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.683005 4191 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:44.683485 4191 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.683763 4191 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P 35294c46c02e422d979a2796c96a5e6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:37:44.690822 4176 tablet_replica.cc:439] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } } }
I20260926 03:37:44.691139 4176 tablet_replica.cc:442] T af9999d0ac194d54a467898818249036 P e3decc86327b4ef994f72dcc7d3adc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:37:44.692761 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.685379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.1.221.129:45337)
0926 03:37:44.685551 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.685557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.685575 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:37:44.685677 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.685682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.685779 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.685923 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:44.686510 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.687694 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.689389 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.690093 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.690215 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.690823 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.691738 (+ 915us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.691885 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.692351 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.692435 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:37:44.709442 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.703649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42807 (local address 127.1.221.129:33523)
0926 03:37:44.704235 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.704252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.704370 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.704683 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.704686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.704694 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:44.704955 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.704962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.706301 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.706304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.707342 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.707353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.708373 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.708377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.708491 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.708558 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.708967 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.708975 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.709030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.709135 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":264,"threads_started":1}
I20260926 03:37:44.709431 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.703869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33523 (local address 127.1.221.131:42807)
0926 03:37:44.704072 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.704078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.704333 (+ 255us) server_negotiation.cc:407] Connection header received
0926 03:37:44.704486 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.704491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.704575 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.704691 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:44.705093 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.706182 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.707554 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.708206 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.708332 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.708749 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.708878 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.708962 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.709051 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.709127 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:37:44.718801 3960 raft_consensus.cc:1261] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Refusing update from remote peer e3decc86327b4ef994f72dcc7d3adc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:44.718873 4099 raft_consensus.cc:1261] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Refusing update from remote peer e3decc86327b4ef994f72dcc7d3adc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:44.719525 4179 consensus_queue.cc:1035] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:44.719863 4176 consensus_queue.cc:1035] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:44.757004 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.750202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37937 (local address 127.1.221.129:45337)
0926 03:37:44.750373 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:44.750379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:44.751063 (+ 684us) server_negotiation.cc:407] Connection header received
0926 03:37:44.751244 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:44.751249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:44.751331 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:44.751454 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:44.752202 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.752909 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.755004 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.755674 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.755769 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.756391 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.756489 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.756554 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:44.756700 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:37:44.756763 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:37:44.757237 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:44.750269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45337 (local address 127.1.221.131:37937)
0926 03:37:44.750950 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:44.750969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:44.751109 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:44.751619 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:44.751624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:44.751637 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:44.752042 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.752053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.753039 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.753045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.754841 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:44.754857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:44.755959 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:44.755966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:44.756143 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:44.756211 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:44.756586 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:44.756598 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:37:44.756680 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:37:44.756831 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
I20260926 03:37:44.792793 3662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:44.793092 3662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:44.796991 3662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Sending DeleteTablet for 3 replicas of tablet fc3f8bbad9744d51ae43303f54b9f801
I20260926 03:37:44.798154 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41806
I20260926 03:37:44.798277 4079 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57982
I20260926 03:37:44.799212 3801 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f8bbad9744d51ae43303f54b9f801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60974
I20260926 03:37:44.800453 3662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:44.801115 3662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:44.809720 4078 tablet_service.cc:1467] Processing CreateTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (DEFAULT_TABLE table=test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.810249 4078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af6e6f3f6ee42b5860d40fd1fc94280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.810423 3800 tablet_service.cc:1467] Processing CreateTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (DEFAULT_TABLE table=test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.810958 3800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af6e6f3f6ee42b5860d40fd1fc94280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.815905 3939 tablet_service.cc:1467] Processing CreateTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (DEFAULT_TABLE table=test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:44.816360 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af6e6f3f6ee42b5860d40fd1fc94280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:44.817121 4170 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Bootstrap starting.
I20260926 03:37:44.818634 4170 tablet_bootstrap.cc:654] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.822723 4169 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Bootstrap starting.
I20260926 03:37:44.823459 4168 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Bootstrap starting.
I20260926 03:37:44.824426 4169 tablet_bootstrap.cc:654] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.829732 4168 tablet_bootstrap.cc:654] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:44.831600 4170 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: No bootstrap required, opened a new log
I20260926 03:37:44.831841 4170 ts_tablet_manager.cc:1397] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260926 03:37:44.832747 4170 raft_consensus.cc:348] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.833150 4170 raft_consensus.cc:374] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.833268 4170 raft_consensus.cc:729] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.833647 4170 consensus_queue.cc:260] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.834559 4170 ts_tablet_manager.cc:1428] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:44.835597 4218 tablet_replica.cc:330] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: stopping tablet replica
I20260926 03:37:44.836064 4218 raft_consensus.cc:2229] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:44.836426 4218 raft_consensus.cc:2258] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:44.838673 4218 ts_tablet_manager.cc:1905] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:44.840513 4217 tablet_replica.cc:330] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: stopping tablet replica
I20260926 03:37:44.840734 4219 tablet_replica.cc:330] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: stopping tablet replica
I20260926 03:37:44.841270 4217 raft_consensus.cc:2229] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:44.841825 4219 raft_consensus.cc:2229] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:44.844300 4217 raft_consensus.cc:2258] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:44.842589 4219 raft_consensus.cc:2258] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:44.846817 4219 ts_tablet_manager.cc:1905] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:44.847434 4218 ts_tablet_manager.cc:1918] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:44.847617 4218 log.cc:1196] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/fc3f8bbad9744d51ae43303f54b9f801
I20260926 03:37:44.848165 4218 ts_tablet_manager.cc:1939] T fc3f8bbad9744d51ae43303f54b9f801 P 35294c46c02e422d979a2796c96a5e6b: Deleting consensus metadata
I20260926 03:37:44.850860 4217 ts_tablet_manager.cc:1905] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:44.856696 4219 ts_tablet_manager.cc:1918] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:44.856964 4219 log.cc:1196] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/fc3f8bbad9744d51ae43303f54b9f801
I20260926 03:37:44.857065 4168 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: No bootstrap required, opened a new log
I20260926 03:37:44.857254 4168 ts_tablet_manager.cc:1397] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.004s
I20260926 03:37:44.857676 4219 ts_tablet_manager.cc:1939] T fc3f8bbad9744d51ae43303f54b9f801 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting consensus metadata
I20260926 03:37:44.858016 4168 raft_consensus.cc:348] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.858258 4168 raft_consensus.cc:374] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.858340 4168 raft_consensus.cc:729] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.858538 4217 ts_tablet_manager.cc:1918] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:44.858701 4217 log.cc:1196] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/fc3f8bbad9744d51ae43303f54b9f801
I20260926 03:37:44.858630 4168 consensus_queue.cc:260] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.859614 3648 catalog_manager.cc:4841] TS 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807): tablet fc3f8bbad9744d51ae43303f54b9f801 (table test-table [id=38b3abbc5b8d4bc29147a222ea52b273]) successfully deleted
I20260926 03:37:44.860697 4217 ts_tablet_manager.cc:1939] T fc3f8bbad9744d51ae43303f54b9f801 P dd84cbc20fda4c1da0fc387b0231b711: Deleting consensus metadata
I20260926 03:37:44.860018 4168 ts_tablet_manager.cc:1428] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 03:37:44.861539 3651 catalog_manager.cc:4841] TS e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337): tablet fc3f8bbad9744d51ae43303f54b9f801 (table test-table [id=38b3abbc5b8d4bc29147a222ea52b273]) successfully deleted
I20260926 03:37:44.863056 3651 catalog_manager.cc:4841] TS dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521): tablet fc3f8bbad9744d51ae43303f54b9f801 (table test-table [id=38b3abbc5b8d4bc29147a222ea52b273]) successfully deleted
I20260926 03:37:44.864737 4169 tablet_bootstrap.cc:492] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: No bootstrap required, opened a new log
I20260926 03:37:44.864899 4169 ts_tablet_manager.cc:1397] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.006s
I20260926 03:37:44.865658 4169 raft_consensus.cc:348] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.865919 4169 raft_consensus.cc:374] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:44.865998 4169 raft_consensus.cc:729] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd84cbc20fda4c1da0fc387b0231b711, State: Initialized, Role: FOLLOWER
I20260926 03:37:44.866232 4169 consensus_queue.cc:260] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:44.867344 4169 ts_tablet_manager.cc:1428] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:37:45.294698 4191 raft_consensus.cc:482] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:45.294886 4191 raft_consensus.cc:504] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:45.295524 4191 leader_election.cc:288] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:45.296105 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af6e6f3f6ee42b5860d40fd1fc94280" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" is_pre_election: true
I20260926 03:37:45.296186 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af6e6f3f6ee42b5860d40fd1fc94280" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
I20260926 03:37:45.296339 3826 raft_consensus.cc:2454] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:45.296423 3959 raft_consensus.cc:2454] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:45.296801 4033 leader_election.cc:302] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711; no voters:
I20260926 03:37:45.297067 4191 raft_consensus.cc:2781] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:45.297181 4191 raft_consensus.cc:482] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:45.297281 4191 raft_consensus.cc:3037] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:45.299098 4191 raft_consensus.cc:504] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:45.299656 4191 leader_election.cc:288] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Requested vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:45.300151 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af6e6f3f6ee42b5860d40fd1fc94280" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
I20260926 03:37:45.300217 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af6e6f3f6ee42b5860d40fd1fc94280" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711"
I20260926 03:37:45.300360 3826 raft_consensus.cc:3037] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:45.300426 3959 raft_consensus.cc:3037] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:45.302264 3826 raft_consensus.cc:2454] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:45.302321 3959 raft_consensus.cc:2454] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:45.302670 4033 leader_election.cc:302] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, e3decc86327b4ef994f72dcc7d3adc4a; no voters:
I20260926 03:37:45.302912 4191 raft_consensus.cc:2781] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:45.303258 4191 raft_consensus.cc:686] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 1 LEADER]: Becoming Leader. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Running, Role: LEADER
I20260926 03:37:45.303524 4191 consensus_queue.cc:237] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:45.306015 3663 catalog_manager.cc:5495] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 35294c46c02e422d979a2796c96a5e6b (127.1.221.131). New cstate: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } health_report { overall_health: HEALTHY } } }
I20260926 03:37:45.408237 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:45.400002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.1.221.131:42807)
0926 03:37:45.403762 (+ 3760us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:45.403767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:45.403784 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:37:45.403841 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:45.403844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:45.403906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:45.404001 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:45.404482 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.405202 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.406537 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.406955 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.407063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:45.407515 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:45.407624 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:45.407709 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:45.407870 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 03:37:45.407948 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3639,"thread_start_us":3533,"threads_started":1}
I20260926 03:37:45.420326 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:45.415930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:42875 (local address 127.1.221.131:42807)
0926 03:37:45.416065 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:45.416069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:45.416443 (+ 374us) server_negotiation.cc:407] Connection header received
0926 03:37:45.416571 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:45.416574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:45.416632 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:45.416700 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:45.417069 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.417742 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.418981 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.419348 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.419478 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:45.419801 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:45.419891 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:45.419951 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:45.420041 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:37:45.420098 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:37:45.420399 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:45.415853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42807 (local address 127.1.221.131:42875)
0926 03:37:45.416362 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:45.416377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:45.416482 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:45.416720 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:45.416723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:45.416731 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:37:45.416954 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:45.416960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.417837 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.417841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:45.418859 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:45.418870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:45.419510 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:45.419521 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:45.419652 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:45.419687 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:37:45.419961 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:37:45.419968 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:37:45.420018 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 03:37:45.420113 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":256,"threads_started":1}
I20260926 03:37:45.427946 3959 raft_consensus.cc:1261] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:45.428030 3826 raft_consensus.cc:1261] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:45.428462 4239 consensus_queue.cc:1035] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:45.428810 4191 consensus_queue.cc:1035] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:37:45.449683 4227 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:37:47.042752 3663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:47.042971 3663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:47.047065 3663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Sending DeleteTablet for 3 replicas of tablet 9af6e6f3f6ee42b5860d40fd1fc94280
I20260926 03:37:47.047911 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41806
I20260926 03:37:47.047940 4079 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57982
I20260926 03:37:47.047940 3801 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af6e6f3f6ee42b5860d40fd1fc94280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60974
I20260926 03:37:47.056195 4268 tablet_replica.cc:330] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: stopping tablet replica
I20260926 03:37:47.056444 4268 raft_consensus.cc:2229] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:47.056736 4268 raft_consensus.cc:2258] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:47.060148 4266 tablet_replica.cc:330] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: stopping tablet replica
I20260926 03:37:47.060338 4268 ts_tablet_manager.cc:1905] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.060431 4267 tablet_replica.cc:330] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: stopping tablet replica
I20260926 03:37:47.060606 4267 raft_consensus.cc:2229] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:47.061038 4267 raft_consensus.cc:2258] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:47.061345 4266 raft_consensus.cc:2229] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:47.061582 4266 raft_consensus.cc:2258] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:47.062274 4267 ts_tablet_manager.cc:1905] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.064033 4266 ts_tablet_manager.cc:1905] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.066663 4268 ts_tablet_manager.cc:1918] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:47.066846 4268 log.cc:1196] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/9af6e6f3f6ee42b5860d40fd1fc94280
I20260926 03:37:47.067270 4268 ts_tablet_manager.cc:1939] T 9af6e6f3f6ee42b5860d40fd1fc94280 P dd84cbc20fda4c1da0fc387b0231b711: Deleting consensus metadata
I20260926 03:37:47.068081 4267 ts_tablet_manager.cc:1918] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:47.068265 4267 log.cc:1196] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/9af6e6f3f6ee42b5860d40fd1fc94280
I20260926 03:37:47.068437 3651 catalog_manager.cc:4841] TS dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521): tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (table test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]) successfully deleted
I20260926 03:37:47.068818 4267 ts_tablet_manager.cc:1939] T 9af6e6f3f6ee42b5860d40fd1fc94280 P 35294c46c02e422d979a2796c96a5e6b: Deleting consensus metadata
I20260926 03:37:47.069921 3648 catalog_manager.cc:4841] TS 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807): tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (table test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]) successfully deleted
I20260926 03:37:47.070665 4266 ts_tablet_manager.cc:1918] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:47.070781 4266 log.cc:1196] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/9af6e6f3f6ee42b5860d40fd1fc94280
I20260926 03:37:47.071172 4266 ts_tablet_manager.cc:1939] T 9af6e6f3f6ee42b5860d40fd1fc94280 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting consensus metadata
I20260926 03:37:47.072216 3651 catalog_manager.cc:4841] TS e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337): tablet 9af6e6f3f6ee42b5860d40fd1fc94280 (table test-table [id=2d8b1f11f7fd4ed19704e6995e33b736]) successfully deleted
Sep 26 03:37:47 dist-test-slave-lqrk krb5kdc[3619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:37:47 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:47.100172 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:47.086966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.1.221.190:33571)
0926 03:37:47.087552 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:47.087556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:47.087578 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:37:47.087749 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:47.087753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:47.087825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:47.087925 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:47.089248 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.090096 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.091296 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.091746 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.091839 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:47.095502 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:47.095508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:47.095511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:47.095551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:47.097989 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:47.098587 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:47.098591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:47.098593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:47.098709 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:47.098969 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:47.098973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:47.098975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:47.099237 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:47.099438 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:47.099665 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 03:37:47.099838 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
I20260926 03:37:47.103510 3663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:37:47.104105 3663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:47.110771 3940 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 (DEFAULT_TABLE table=test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.111227 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbeabb6257a46a895908bc4fb8cc954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.111132 3801 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 (DEFAULT_TABLE table=test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.111574 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbeabb6257a46a895908bc4fb8cc954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.115857 4277 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Bootstrap starting.
I20260926 03:37:47.116667 4079 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 (DEFAULT_TABLE table=test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.117080 4079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbeabb6257a46a895908bc4fb8cc954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.117115 4278 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Bootstrap starting.
I20260926 03:37:47.117627 4277 tablet_bootstrap.cc:654] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.118350 4278 tablet_bootstrap.cc:654] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.120508 4278 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: No bootstrap required, opened a new log
I20260926 03:37:47.120639 4278 ts_tablet_manager.cc:1397] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:37:47.121217 4278 raft_consensus.cc:348] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.121407 4278 raft_consensus.cc:374] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.121487 4278 raft_consensus.cc:729] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.121976 4278 consensus_queue.cc:260] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.122278 4281 raft_consensus.cc:482] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:47.122512 4281 raft_consensus.cc:504] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.123893 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
W20260926 03:37:47.124987 3754 leader_election.cc:341] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521): Illegal state: must be running to vote when last-logged opid is not known
I20260926 03:37:47.125753 4278 ts_tablet_manager.cc:1428] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 03:37:47.125891 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35294c46c02e422d979a2796c96a5e6b" is_pre_election: true
I20260926 03:37:47.126578 4281 leader_election.cc:288] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521), 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
W20260926 03:37:47.126695 3751 leader_election.cc:341] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807): Illegal state: must be running to vote when last-logged opid is not known
I20260926 03:37:47.126828 3751 leader_election.cc:302] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3decc86327b4ef994f72dcc7d3adc4a; no voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711
I20260926 03:37:47.127086 4281 raft_consensus.cc:2726] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:37:47.127152 4277 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: No bootstrap required, opened a new log
I20260926 03:37:47.127285 4277 ts_tablet_manager.cc:1397] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260926 03:37:47.127831 4283 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Bootstrap starting.
I20260926 03:37:47.128069 4277 raft_consensus.cc:348] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.128306 4277 raft_consensus.cc:374] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.128396 4277 raft_consensus.cc:729] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd84cbc20fda4c1da0fc387b0231b711, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.128659 4277 consensus_queue.cc:260] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.129431 4277 ts_tablet_manager.cc:1428] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:37:47.129467 4283 tablet_bootstrap.cc:654] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.131855 4283 tablet_bootstrap.cc:492] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: No bootstrap required, opened a new log
I20260926 03:37:47.131978 4283 ts_tablet_manager.cc:1397] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:37:47.132691 4283 raft_consensus.cc:348] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.132860 4283 raft_consensus.cc:374] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.132936 4283 raft_consensus.cc:729] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.133149 4283 consensus_queue.cc:260] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.133634 4283 ts_tablet_manager.cc:1428] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:37:47.443733 4249 raft_consensus.cc:482] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:47.443941 4249 raft_consensus.cc:504] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.444630 4249 leader_election.cc:288] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:47.445266 3825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" is_pre_election: true
I20260926 03:37:47.445281 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
I20260926 03:37:47.445535 3825 raft_consensus.cc:2454] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:47.445557 3960 raft_consensus.cc:2454] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 0.
I20260926 03:37:47.446007 4033 leader_election.cc:302] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711; no voters:
I20260926 03:37:47.446270 4249 raft_consensus.cc:2781] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:47.446408 4249 raft_consensus.cc:482] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:47.446594 4249 raft_consensus.cc:3037] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.448757 4249 raft_consensus.cc:504] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.449331 4249 leader_election.cc:288] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Requested vote from peers e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337), dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521)
I20260926 03:37:47.449877 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711"
I20260926 03:37:47.449982 3825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbeabb6257a46a895908bc4fb8cc954" candidate_uuid: "35294c46c02e422d979a2796c96a5e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3decc86327b4ef994f72dcc7d3adc4a"
I20260926 03:37:47.450174 3960 raft_consensus.cc:3037] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.450201 3825 raft_consensus.cc:3037] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.452996 3960 raft_consensus.cc:2454] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:47.453013 3825 raft_consensus.cc:2454] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35294c46c02e422d979a2796c96a5e6b in term 1.
I20260926 03:37:47.453485 4033 leader_election.cc:302] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, dd84cbc20fda4c1da0fc387b0231b711; no voters:
I20260926 03:37:47.453769 4249 raft_consensus.cc:2781] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:47.453908 4249 raft_consensus.cc:686] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 1 LEADER]: Becoming Leader. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Running, Role: LEADER
I20260926 03:37:47.454170 4249 consensus_queue.cc:237] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.457021 3663 catalog_manager.cc:5495] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 35294c46c02e422d979a2796c96a5e6b (127.1.221.131). New cstate: current_term: 1 leader_uuid: "35294c46c02e422d979a2796c96a5e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } health_report { overall_health: HEALTHY } } }
I20260926 03:37:47.508718 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:47.503773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.1.221.131:42807)
0926 03:37:47.504341 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:47.504345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:47.504364 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:37:47.504416 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:47.504419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:47.504484 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:47.504567 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:47.504892 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.505653 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.506940 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.507439 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.507550 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:47.507928 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:47.508049 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:47.508117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:47.508295 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:37:47.508361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":333,"threads_started":1}
I20260926 03:37:47.517100 3960 raft_consensus.cc:1261] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:47.517105 3825 raft_consensus.cc:1261] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Refusing update from remote peer 35294c46c02e422d979a2796c96a5e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:47.517603 4249 consensus_queue.cc:1035] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:47.517899 4245 consensus_queue.cc:1035] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:47.569644 3663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:47.569875 3663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:47.572986 3663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Sending DeleteTablet for 3 replicas of tablet 8cbeabb6257a46a895908bc4fb8cc954
I20260926 03:37:47.573925 3801 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60974
I20260926 03:37:47.573969 4078 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57982
I20260926 03:37:47.574337 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbeabb6257a46a895908bc4fb8cc954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41806
I20260926 03:37:47.574545 4294 tablet_replica.cc:330] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: stopping tablet replica
I20260926 03:37:47.574735 4294 raft_consensus.cc:2229] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:47.574975 4295 tablet_replica.cc:330] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: stopping tablet replica
I20260926 03:37:47.575163 4294 raft_consensus.cc:2258] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:47.574896 3663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:37:47.575513 4295 raft_consensus.cc:2229] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:47.575779 4295 raft_consensus.cc:2258] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 03:37:47.576242 3663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:37:47.576619 4294 ts_tablet_manager.cc:1905] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.576862 4295 ts_tablet_manager.cc:1905] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.583323 4295 ts_tablet_manager.cc:1918] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:47.583320 4294 ts_tablet_manager.cc:1918] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:47.583575 4294 log.cc:1196] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/8cbeabb6257a46a895908bc4fb8cc954
I20260926 03:37:47.583598 4295 log.cc:1196] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/8cbeabb6257a46a895908bc4fb8cc954
I20260926 03:37:47.583995 4294 ts_tablet_manager.cc:1939] T 8cbeabb6257a46a895908bc4fb8cc954 P 35294c46c02e422d979a2796c96a5e6b: Deleting consensus metadata
I20260926 03:37:47.584259 4295 ts_tablet_manager.cc:1939] T 8cbeabb6257a46a895908bc4fb8cc954 P dd84cbc20fda4c1da0fc387b0231b711: Deleting consensus metadata
I20260926 03:37:47.584540 3940 tablet_service.cc:1467] Processing CreateTablet for tablet ff93c73811cb41d88bf9a861f592f284 (DEFAULT_TABLE table=test-table [id=78ebd015596448bc9682655b025e1d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.584985 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff93c73811cb41d88bf9a861f592f284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.585527 3651 catalog_manager.cc:4841] TS dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521): tablet 8cbeabb6257a46a895908bc4fb8cc954 (table test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]) successfully deleted
I20260926 03:37:47.587044 4293 tablet_replica.cc:330] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: stopping tablet replica
I20260926 03:37:47.587246 4293 raft_consensus.cc:2229] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:47.587482 4293 raft_consensus.cc:2258] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:47.588227 3648 catalog_manager.cc:4841] TS 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807): tablet 8cbeabb6257a46a895908bc4fb8cc954 (table test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]) successfully deleted
I20260926 03:37:47.588232 3800 tablet_service.cc:1467] Processing CreateTablet for tablet ff93c73811cb41d88bf9a861f592f284 (DEFAULT_TABLE table=test-table [id=78ebd015596448bc9682655b025e1d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.588620 4293 ts_tablet_manager.cc:1905] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:47.588953 4078 tablet_service.cc:1467] Processing CreateTablet for tablet ff93c73811cb41d88bf9a861f592f284 (DEFAULT_TABLE table=test-table [id=78ebd015596448bc9682655b025e1d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:37:47.589349 4078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff93c73811cb41d88bf9a861f592f284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.591576 4277 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Bootstrap starting.
I20260926 03:37:47.593116 4277 tablet_bootstrap.cc:654] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.594884 4283 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Bootstrap starting.
I20260926 03:37:47.588639 3800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff93c73811cb41d88bf9a861f592f284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:47.595453 4277 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: No bootstrap required, opened a new log
I20260926 03:37:47.595619 4277 ts_tablet_manager.cc:1397] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:37:47.596381 4283 tablet_bootstrap.cc:654] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.596330 4277 raft_consensus.cc:348] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.596578 4277 raft_consensus.cc:374] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.596642 4277 raft_consensus.cc:729] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd84cbc20fda4c1da0fc387b0231b711, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.597213 4277 consensus_queue.cc:260] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.597755 4277 ts_tablet_manager.cc:1428] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:37:47.599133 4283 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: No bootstrap required, opened a new log
I20260926 03:37:47.599282 4283 ts_tablet_manager.cc:1397] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:37:47.600921 4283 raft_consensus.cc:348] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.601178 4283 raft_consensus.cc:374] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.601246 4283 raft_consensus.cc:729] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35294c46c02e422d979a2796c96a5e6b, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.601468 4283 consensus_queue.cc:260] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.601824 4278 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Bootstrap starting.
I20260926 03:37:47.602056 4283 ts_tablet_manager.cc:1428] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:47.603426 4278 tablet_bootstrap.cc:654] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:47.603965 4293 ts_tablet_manager.cc:1918] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:37:47.604094 4293 log.cc:1196] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/8cbeabb6257a46a895908bc4fb8cc954
I20260926 03:37:47.604535 4293 ts_tablet_manager.cc:1939] T 8cbeabb6257a46a895908bc4fb8cc954 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting consensus metadata
I20260926 03:37:47.605825 3651 catalog_manager.cc:4841] TS e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337): tablet 8cbeabb6257a46a895908bc4fb8cc954 (table test-table [id=6c36a1b53eb94ad4bf7fcd59b8e476f1]) successfully deleted
I20260926 03:37:47.608556 4278 tablet_bootstrap.cc:492] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: No bootstrap required, opened a new log
I20260926 03:37:47.608685 4278 ts_tablet_manager.cc:1397] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 03:37:47.609725 4278 raft_consensus.cc:348] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.609958 4278 raft_consensus.cc:374] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:47.610031 4278 raft_consensus.cc:729] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Initialized, Role: FOLLOWER
I20260926 03:37:47.610245 4278 consensus_queue.cc:260] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.611215 4278 ts_tablet_manager.cc:1428] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:37:47.654649 4281 raft_consensus.cc:482] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:37:47.654829 4281 raft_consensus.cc:504] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.655591 4281 leader_election.cc:288] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521), 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
I20260926 03:37:47.656090 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff93c73811cb41d88bf9a861f592f284" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711" is_pre_election: true
I20260926 03:37:47.656378 3960 raft_consensus.cc:2454] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 0.
I20260926 03:37:47.656411 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff93c73811cb41d88bf9a861f592f284" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35294c46c02e422d979a2796c96a5e6b" is_pre_election: true
I20260926 03:37:47.656684 4099 raft_consensus.cc:2454] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 0.
I20260926 03:37:47.656819 3754 leader_election.cc:302] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd84cbc20fda4c1da0fc387b0231b711, e3decc86327b4ef994f72dcc7d3adc4a; no voters:
I20260926 03:37:47.657127 4281 raft_consensus.cc:2781] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:37:47.657248 4281 raft_consensus.cc:482] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:37:47.657327 4281 raft_consensus.cc:3037] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.659570 4281 raft_consensus.cc:504] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.660202 4281 leader_election.cc:288] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 election: Requested vote from peers dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521), 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807)
I20260926 03:37:47.660633 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff93c73811cb41d88bf9a861f592f284" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd84cbc20fda4c1da0fc387b0231b711"
I20260926 03:37:47.660674 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff93c73811cb41d88bf9a861f592f284" candidate_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35294c46c02e422d979a2796c96a5e6b"
I20260926 03:37:47.660847 3960 raft_consensus.cc:3037] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.660880 4099 raft_consensus.cc:3037] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:47.663095 4099 raft_consensus.cc:2454] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 1.
I20260926 03:37:47.663585 3751 leader_election.cc:302] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35294c46c02e422d979a2796c96a5e6b, e3decc86327b4ef994f72dcc7d3adc4a; no voters:
I20260926 03:37:47.663868 4281 raft_consensus.cc:2781] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:47.663892 3960 raft_consensus.cc:2454] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3decc86327b4ef994f72dcc7d3adc4a in term 1.
I20260926 03:37:47.669441 4281 raft_consensus.cc:686] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 LEADER]: Becoming Leader. State: Replica: e3decc86327b4ef994f72dcc7d3adc4a, State: Running, Role: LEADER
I20260926 03:37:47.669740 4281 consensus_queue.cc:237] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } }
I20260926 03:37:47.672806 3663 catalog_manager.cc:5495] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129). New cstate: current_term: 1 leader_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3decc86327b4ef994f72dcc7d3adc4a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 } health_report { overall_health: UNKNOWN } } }
I20260926 03:37:47.694906 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:47.689739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.1.221.129:45337)
0926 03:37:47.690314 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:47.690319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:47.690335 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:37:47.690384 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:47.690387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:47.690449 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:47.690530 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:47.690960 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.691889 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.693227 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:47.693668 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:47.693787 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:47.694218 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:47.694319 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:47.694391 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:47.694537 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:37:47.694610 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
I20260926 03:37:47.703454 3960 raft_consensus.cc:1261] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Refusing update from remote peer e3decc86327b4ef994f72dcc7d3adc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:47.703565 4099 raft_consensus.cc:1261] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Refusing update from remote peer e3decc86327b4ef994f72dcc7d3adc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:47.703962 4299 consensus_queue.cc:1035] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd84cbc20fda4c1da0fc387b0231b711" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:47.704298 4281 consensus_queue.cc:1035] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35294c46c02e422d979a2796c96a5e6b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:37:47.723799 4252 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:37:49.302194 3664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:49.302431 3664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:37:49.306967 3664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 539255bb553e43f2a1dc08c7ccfeac64: Sending DeleteTablet for 3 replicas of tablet ff93c73811cb41d88bf9a861f592f284
I20260926 03:37:49.307744 3800 tablet_service.cc:1514] Processing DeleteTablet for tablet ff93c73811cb41d88bf9a861f592f284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:49 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60974
I20260926 03:37:49.307904 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet ff93c73811cb41d88bf9a861f592f284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:49 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41806
I20260926 03:37:49.307950 4078 tablet_service.cc:1514] Processing DeleteTablet for tablet ff93c73811cb41d88bf9a861f592f284 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:37:49 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57982
I20260926 03:37:49.308527 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3735
I20260926 03:37:49.318923 4335 tablet_replica.cc:330] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: stopping tablet replica
I20260926 03:37:49.319406 4335 raft_consensus.cc:2229] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:49.319782 4335 raft_consensus.cc:2258] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:49.320448 4333 tablet_replica.cc:330] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: stopping tablet replica
I20260926 03:37:49.321357 4335 ts_tablet_manager.cc:1905] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:49.322113 4333 raft_consensus.cc:2229] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:37:49.322575 4333 raft_consensus.cc:2258] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:49.323701 4333 ts_tablet_manager.cc:1905] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:49.329478 4333 ts_tablet_manager.cc:1918] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:49.329480 4335 ts_tablet_manager.cc:1918] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:49.329607 4333 log.cc:1196] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/ff93c73811cb41d88bf9a861f592f284
I20260926 03:37:49.329617 4335 log.cc:1196] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/ff93c73811cb41d88bf9a861f592f284
I20260926 03:37:49.329996 4335 ts_tablet_manager.cc:1939] T ff93c73811cb41d88bf9a861f592f284 P dd84cbc20fda4c1da0fc387b0231b711: Deleting consensus metadata
I20260926 03:37:49.330024 4334 tablet_replica.cc:330] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: stopping tablet replica
I20260926 03:37:49.330193 4334 raft_consensus.cc:2229] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:37:49.330399 4334 raft_consensus.cc:2258] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:37:49.330915 4333 ts_tablet_manager.cc:1939] T ff93c73811cb41d88bf9a861f592f284 P e3decc86327b4ef994f72dcc7d3adc4a: Deleting consensus metadata
I20260926 03:37:49.331102 3651 catalog_manager.cc:4841] TS dd84cbc20fda4c1da0fc387b0231b711 (127.1.221.130:41521): tablet ff93c73811cb41d88bf9a861f592f284 (table test-table [id=78ebd015596448bc9682655b025e1d53]) successfully deleted
I20260926 03:37:49.331434 4334 ts_tablet_manager.cc:1905] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:37:49.331998 3651 catalog_manager.cc:4841] TS e3decc86327b4ef994f72dcc7d3adc4a (127.1.221.129:45337): tablet ff93c73811cb41d88bf9a861f592f284 (table test-table [id=78ebd015596448bc9682655b025e1d53]) successfully deleted
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:49.336548 4334 ts_tablet_manager.cc:1918] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:37:49.336719 4334 log.cc:1196] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/ff93c73811cb41d88bf9a861f592f284
I20260926 03:37:49.337157 4334 ts_tablet_manager.cc:1939] T ff93c73811cb41d88bf9a861f592f284 P 35294c46c02e422d979a2796c96a5e6b: Deleting consensus metadata
I20260926 03:37:49.338199 3648 catalog_manager.cc:4841] TS 35294c46c02e422d979a2796c96a5e6b (127.1.221.131:42807): tablet ff93c73811cb41d88bf9a861f592f284 (table test-table [id=78ebd015596448bc9682655b025e1d53]) successfully deleted
I20260926 03:37:49.343030 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:49.310050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.1.221.129:45337)
0926 03:37:49.321543 (+ 11493us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:49.321549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:49.321573 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:37:49.321625 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:49.321629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:49.321691 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:49.321787 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:49.323349 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.324382 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.326178 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.326856 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.326972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:49.337750 (+ 10778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:49.337766 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:49.337782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:49.337824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:49.340723 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.341346 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.341352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.341354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.341480 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.341756 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.341759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.341761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.342094 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:49.342302 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:49.342534 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:37:49.342709 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11370,"thread_start_us":11256,"threads_started":1}
W20260926 03:37:49.343667 3866 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32776
W20260926 03:37:49.344214 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:49.344413 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3735
I20260926 03:37:49.362027 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3875
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:37:49.375824 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:49.362782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.1.221.130:41521)
0926 03:37:49.363263 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:49.363268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:49.363286 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:37:49.363338 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:49.363342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:49.363445 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:49.363546 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:49.364697 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.365435 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.366579 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.366943 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.367025 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:49.370954 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:49.370969 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:49.370985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:49.371026 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:49.373596 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.374187 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.374194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.374197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.374325 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.374602 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.374605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.374607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.374934 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:49.375141 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:49.375344 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 03:37:49.375568 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
W20260926 03:37:49.376493 4005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41812
W20260926 03:37:49.377146 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:49.377288 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3875
I20260926 03:37:49.390954 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4014
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[3619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:37:49.406104 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:49.391819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.1.221.131:42807)
0926 03:37:49.392297 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:49.392301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:49.392317 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:37:49.392370 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:49.392374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:49.392431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:49.392524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:49.393697 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.394411 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.395816 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:49.396268 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:49.396363 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:49.400798 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:49.400815 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:49.400831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:49.400867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:49.403812 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.404410 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.404417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.404419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.404539 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:49.404827 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:49.404830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:49.404832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:49.405172 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:49.405393 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:49.405586 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:37:49.405751 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
W20260926 03:37:49.406816 4143 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58022
W20260926 03:37:49.407487 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:49.407644 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4014
I20260926 03:37:49.427511 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 3632
I20260926 03:37:49.428689 3704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:49.542613 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 3632
2026-09-26T03:37:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8621 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[4343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[4343](info): set up 2 sockets
Sep 26 03:37:49 dist-test-slave-lqrk krb5kdc[4343](info): commencing operation
krb5kdc: starting...
W20260926 03:37:50.563868 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:50 dist-test-slave-lqrk krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:50.741055 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33349
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:51.056396 4356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:51.056717 4356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:51.056855 4356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:51.064208 4356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:51.064348 4356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:51.064435 4356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:51.064515 4356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:51.064576 4356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:51.074895 4356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:51.076331 4356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:51.077662 4356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:51.089432 4361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:51.089390 4362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:51.089450 4364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:51.090128 4356 server_base.cc:1034] running on GCE node
I20260926 03:37:51.090977 4356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:51.092500 4356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:51.093748 4356 hybrid_clock.cc:648] HybridClock initialized: now 1790393871093682 us; error 52 us; skew 500 ppm
I20260926 03:37:51.094151 4356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:51 dist-test-slave-lqrk krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393871, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:51.097496 4356 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:51.099116 4356 webserver.cc:460] Webserver started at http://127.1.221.190:38895/ using document root <none> and password file <none>
I20260926 03:37:51.099889 4356 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:51.100024 4356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:51.100382 4356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:51.103742 4356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "cc627e757e5a46bca50f481246140bd9"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "9b17bc09e92640f1127b090a4f6ee84b"
server_key_iv: "de015452181ab9c59fc364a23eee3fae"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.104632 4356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "cc627e757e5a46bca50f481246140bd9"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "9b17bc09e92640f1127b090a4f6ee84b"
server_key_iv: "de015452181ab9c59fc364a23eee3fae"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.110526 4356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:51.114403 4371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:51.116010 4356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:51.116199 4356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "cc627e757e5a46bca50f481246140bd9"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "9b17bc09e92640f1127b090a4f6ee84b"
server_key_iv: "de015452181ab9c59fc364a23eee3fae"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.116400 4356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:51.147675 4356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:51.152074 4356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:51.152349 4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:51.169219 4356 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33349
I20260926 03:37:51.169241 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33349 every 8 connection(s)
I20260926 03:37:51.170581 4356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:51.174374 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:51.178187 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4356
I20260926 03:37:51.178428 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:51.179174 1910 external_mini_cluster.cc:1428] Setting key b13d9623c30c6adb385123206544c261
I20260926 03:37:51.187345 4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9: Bootstrap starting.
I20260926 03:37:51.190438 4423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:51.191438 4423 log.cc:824] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:37:51 dist-test-slave-lqrk krb5kdc[4343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:51.194391 4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9: No bootstrap required, opened a new log
I20260926 03:37:51.202448 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:51.181286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.1.221.190:33349)
0926 03:37:51.182185 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:51.182198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:51.182243 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:37:51.183927 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:51.183951 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:51.184398 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:51.184819 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:51.186231 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.187632 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.188855 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.189549 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.189713 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:51.193713 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:51.193722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:51.193738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:51.193778 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:51.197590 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:51.198398 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:51.198407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:51.198416 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:51.198650 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:51.199043 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:51.199048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:51.199052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:51.199803 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:51.200117 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:51.200525 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:37:51.201019 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":346,"threads_started":1}
I20260926 03:37:51.204334 4423 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } }
I20260926 03:37:51.204694 4423 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:51.204793 4423 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc627e757e5a46bca50f481246140bd9, State: Initialized, Role: FOLLOWER
I20260926 03:37:51.205408 4423 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } }
I20260926 03:37:51.205643 4423 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:51.205798 4423 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:51.205947 4423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:51.208077 4423 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } }
I20260926 03:37:51.208575 4423 leader_election.cc:302] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc627e757e5a46bca50f481246140bd9; no voters:
I20260926 03:37:51.209167 4423 leader_election.cc:288] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:51.209244 4428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:51.209911 4428 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [term 1 LEADER]: Becoming Leader. State: Replica: cc627e757e5a46bca50f481246140bd9, State: Running, Role: LEADER
I20260926 03:37:51.210428 4428 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } }
I20260926 03:37:51.211128 4423 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:51.213195 4429 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc627e757e5a46bca50f481246140bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } } }
I20260926 03:37:51.213601 4429 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:51.213443 4430 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc627e757e5a46bca50f481246140bd9. Latest consensus state: current_term: 1 leader_uuid: "cc627e757e5a46bca50f481246140bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc627e757e5a46bca50f481246140bd9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33349 } } }
I20260926 03:37:51.213727 4430 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:51.216159 4437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:51.221206 4437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:51.230741 4437 catalog_manager.cc:1347] Generated new cluster ID: 081417bcf89b45d592ecd5fbe5376f5e
I20260926 03:37:51.230890 4437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:51.244573 4437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:51.245930 4437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:51.252103 4437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc627e757e5a46bca50f481246140bd9: Generated new TSK 0
I20260926 03:37:51.252907 4437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:51.307371 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:51.624282 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:51.624594 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:51.624727 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:51.632241 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:51.632395 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:51.632568 4449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:51.642850 4449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:51.644277 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:51.645604 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:51.658907 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:51.658872 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:51.658846 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:51.659621 4449 server_base.cc:1034] running on GCE node
I20260926 03:37:51.660302 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:51.661296 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:51.662545 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790393871662478 us; error 55 us; skew 500 ppm
I20260926 03:37:51.662943 4449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:51 dist-test-slave-lqrk krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393871, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:51.666445 4449 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:51.668020 4449 webserver.cc:460] Webserver started at http://127.1.221.129:42185/ using document root <none> and password file <none>
I20260926 03:37:51.668754 4449 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:51.668879 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:51.669183 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:51.672200 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "6781c63197ef4a608c3d30de2513420c"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "099ad862a930d530951194d51e154ae5"
server_key_iv: "0c2f1906cc071efb568e92fa7c6cc732"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.672987 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "6781c63197ef4a608c3d30de2513420c"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "099ad862a930d530951194d51e154ae5"
server_key_iv: "0c2f1906cc071efb568e92fa7c6cc732"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.678475 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:37:51.682086 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:51.683570 4449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:51.683722 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "6781c63197ef4a608c3d30de2513420c"
format_stamp: "Formatted at 2026-09-26 03:37:51 on dist-test-slave-lqrk"
server_key: "099ad862a930d530951194d51e154ae5"
server_key_iv: "0c2f1906cc071efb568e92fa7c6cc732"
server_key_version: "encryptionkey@0"
I20260926 03:37:51.683919 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:51.697918 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:51.702589 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:51.702898 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:51.703989 4449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:51.705430 4449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:51.705514 4449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:51.705648 4449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:51.705713 4449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:51.738024 4449 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:45777
I20260926 03:37:51.738054 4576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:45777 every 8 connection(s)
I20260926 03:37:51.739419 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:51.746718 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4449
I20260926 03:37:51.746944 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:51.747637 1910 external_mini_cluster.cc:1428] Setting key 23b0f248831aff1abf3bbeff343f60cf
Sep 26 03:37:51 dist-test-slave-lqrk krb5kdc[4343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393871, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:51.759438 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:51.741930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53719 (local address 127.1.221.190:33349)
0926 03:37:51.742335 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:51.742340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:51.743347 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 03:37:51.745188 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:51.745193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:51.745279 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:51.745386 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:51.747153 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.747967 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.749817 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.750208 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.750302 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:51.754200 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:51.754205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:51.754209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:51.754239 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:51.756621 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:51.757283 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:51.757289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:51.757291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:51.757409 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:51.757790 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:51.757795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:51.757799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:51.758061 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:51.758193 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:51.758843 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 03:37:51.759056 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":197,"threads_started":1}
I20260926 03:37:51.760483 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:51.742465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33349 (local address 127.1.221.129:53719)
0926 03:37:51.743171 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:51.743216 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:51.744913 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:51.745628 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:51.745637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:51.746168 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:51.746969 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:51.746999 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.748121 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.748126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:51.749685 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:51.749699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:51.750416 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:51.750422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:51.750711 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:51.751459 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:51.751488 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:51.753971 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:51.756754 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:51.756763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:51.756780 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:51.757176 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:51.757507 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:51.757512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:51.757516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:51.757692 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:51.758180 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:51.758187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:51.758518 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 03:37:51.758754 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 03:37:51.759276 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":261,"threads_started":1}
I20260926 03:37:51.762192 4577 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33349
I20260926 03:37:51.762532 4577 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:51.763248 4577 heartbeater.cc:511] Master 127.1.221.190:33349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:37:51.765334 4388 ts_manager.cc:194] Registered new tserver with Master: 6781c63197ef4a608c3d30de2513420c (127.1.221.129:45777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:37:51.766805 4388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:53719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:51.788636 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:52.098708 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:52.098999 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:52.099119 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:52.106597 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:52.106716 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:52.106868 4583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:52.117082 4583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:52.118366 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:52.119735 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:52.134554 4588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:52.134614 4589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:52.134546 4591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:52.135582 4583 server_base.cc:1034] running on GCE node
I20260926 03:37:52.136248 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:52.137210 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:52.138453 4583 hybrid_clock.cc:648] HybridClock initialized: now 1790393872138380 us; error 57 us; skew 500 ppm
I20260926 03:37:52.138792 4583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393872, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:52.141908 4583 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:52.143294 4583 webserver.cc:460] Webserver started at http://127.1.221.130:33305/ using document root <none> and password file <none>
I20260926 03:37:52.144014 4583 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:52.144129 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:52.144413 4583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:52.147217 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "93514f54ac534f5b87aa546e512f3830"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "cc62311ee3a5aabf507ff98b09cb404e"
server_key_iv: "c419a90cbc05c95e7297810b92dde2cc"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.148013 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "93514f54ac534f5b87aa546e512f3830"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "cc62311ee3a5aabf507ff98b09cb404e"
server_key_iv: "c419a90cbc05c95e7297810b92dde2cc"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.152927 4583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:37:52.156288 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.157618 4583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:52.157756 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "93514f54ac534f5b87aa546e512f3830"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "cc62311ee3a5aabf507ff98b09cb404e"
server_key_iv: "c419a90cbc05c95e7297810b92dde2cc"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.157927 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:52.175850 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:52.179955 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:52.180224 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:52.181191 4583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:52.182618 4583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:52.182690 4583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.182793 4583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:52.182835 4583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.211743 4583 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38777
I20260926 03:37:52.211773 4710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38777 every 8 connection(s)
I20260926 03:37:52.213019 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:52.215265 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4583
I20260926 03:37:52.215489 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:52.215930 1910 external_mini_cluster.cc:1428] Setting key e6481b34c98f80957a55d3a123e16a64
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393872, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:37:52.234272 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:52.215534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41179 (local address 127.1.221.190:33349)
0926 03:37:52.215733 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:52.215740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:52.217021 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 03:37:52.218892 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:52.218898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:52.218990 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:52.219099 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:52.220801 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.221783 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.223835 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.224223 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.224317 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:52.228353 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:52.228359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:52.228362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:52.228399 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:52.230944 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:52.231916 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:52.231921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:52.231923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:52.232067 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:52.232577 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:52.232581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:52.232583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:52.232843 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:52.232975 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:52.233834 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 03:37:52.234029 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:37:52.235474 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:52.216058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33349 (local address 127.1.221.130:41179)
0926 03:37:52.216866 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:52.216914 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:52.218606 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:52.219303 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:52.219312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:52.219966 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:52.220608 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:52.220626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.221952 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.221956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:52.223683 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:52.223698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.224452 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.224459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:52.224735 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:52.225404 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:52.225447 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:52.228114 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:52.231102 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:52.231112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:52.231133 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:52.231785 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:52.232178 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:52.232183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:52.232187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:52.232450 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:52.232970 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:52.232978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:52.233451 (+ 473us) client_negotiation.cc:769] Sending connection context
0926 03:37:52.233742 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 03:37:52.234159 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":311,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:37:52.237093 4711 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33349
I20260926 03:37:52.237432 4711 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:52.238282 4711 heartbeater.cc:511] Master 127.1.221.190:33349 requested a full tablet report, sending...
I20260926 03:37:52.239981 4388 ts_manager.cc:194] Registered new tserver with Master: 93514f54ac534f5b87aa546e512f3830 (127.1.221.130:38777)
I20260926 03:37:52.240830 4388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:41179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:52.258313 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:52.571128 4716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:52.571445 4716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:52.571573 4716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:52.578787 4716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:52.578908 4716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:52.579022 4716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:52.589131 4716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:52.590382 4716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:52.591704 4716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:52.604844 4724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:52.604928 4722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:52.605104 4721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:52.605839 4716 server_base.cc:1034] running on GCE node
I20260926 03:37:52.606429 4716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:52.607465 4716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:52.608672 4716 hybrid_clock.cc:648] HybridClock initialized: now 1790393872608626 us; error 42 us; skew 500 ppm
I20260926 03:37:52.608985 4716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393872, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:52.612171 4716 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:52.613745 4716 webserver.cc:460] Webserver started at http://127.1.221.131:45047/ using document root <none> and password file <none>
I20260926 03:37:52.614486 4716 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:52.614598 4716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:52.614889 4716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:52.617991 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "61215ade30554c719fbc4117203dbf48"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "9fad14ca25dae22fd751f729e0b9882d"
server_key_iv: "1ec14eac8e9d6630cfe372a91530a3aa"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.618722 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "61215ade30554c719fbc4117203dbf48"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "9fad14ca25dae22fd751f729e0b9882d"
server_key_iv: "1ec14eac8e9d6630cfe372a91530a3aa"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.624049 4716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:37:52.627640 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.628973 4716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:37:52.629110 4716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "61215ade30554c719fbc4117203dbf48"
format_stamp: "Formatted at 2026-09-26 03:37:52 on dist-test-slave-lqrk"
server_key: "9fad14ca25dae22fd751f729e0b9882d"
server_key_iv: "1ec14eac8e9d6630cfe372a91530a3aa"
server_key_version: "encryptionkey@0"
I20260926 03:37:52.629277 4716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:52.647279 4716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:52.651592 4716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:52.651849 4716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:52.652805 4716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:52.654182 4716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:52.654253 4716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.654357 4716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:52.654407 4716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:52.684666 4716 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36467
I20260926 03:37:52.684713 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36467 every 8 connection(s)
I20260926 03:37:52.685935 4716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:52.695814 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4716
I20260926 03:37:52.695973 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:52.696354 1910 external_mini_cluster.cc:1428] Setting key b5873ee00ff0c805fd7bdd03ca93a207
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393872, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:52.706334 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:52.688588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:45979 (local address 127.1.221.190:33349)
0926 03:37:52.688808 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:52.688813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:52.690044 (+ 1231us) server_negotiation.cc:407] Connection header received
0926 03:37:52.692004 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:52.692011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:52.692099 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:52.692201 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:52.693833 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.694602 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.696389 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.696934 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.697066 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:52.700869 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:52.700874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:52.700878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:52.700917 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:52.703432 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:52.704127 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:52.704131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:52.704134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:52.704250 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:52.704666 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:52.704671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:52.704673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:52.704952 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:52.705079 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:52.705759 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 03:37:52.705986 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:37:52.707281 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:52.689117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33349 (local address 127.1.221.131:45979)
0926 03:37:52.689855 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:52.689907 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:52.691695 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:52.692464 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:52.692474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:52.693061 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:52.693646 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:52.693664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.694780 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.694785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:52.696235 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:52.696248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:52.697175 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:52.697184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:52.697459 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:52.698181 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:52.698221 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:52.700635 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:52.703562 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:52.703572 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:52.703591 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:52.704001 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:52.704357 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:52.704363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:52.704366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:52.704557 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:52.705071 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:52.705079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:52.705415 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 03:37:52.705653 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 03:37:52.706177 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":271,"threads_started":1}
I20260926 03:37:52.708612 4844 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33349
I20260926 03:37:52.708871 4844 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:52.709537 4844 heartbeater.cc:511] Master 127.1.221.190:33349 requested a full tablet report, sending...
I20260926 03:37:52.711160 4388 ts_manager.cc:194] Registered new tserver with Master: 61215ade30554c719fbc4117203dbf48 (127.1.221.131:36467)
I20260926 03:37:52.712004 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:37:52.712172 4388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:45979
W20260926 03:37:52.740156 4579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:52.737477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.1.221.190:33349)
0926 03:37:52.737676 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:52.737682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:52.738125 (+ 443us) server_negotiation.cc:407] Connection header received
0926 03:37:52.738322 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:52.738327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:52.738392 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:52.738475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:52.739862 (+ 1387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59852: BlockingRecv error: recv got EOF from 127.0.0.1:59852 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:37:52.744169 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4449
W20260926 03:37:52.746835 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4449: Not authorized: Client connection negotiation failed: client connection to 127.1.221.129:45777: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:37:52.747022 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4449
I20260926 03:37:52.759166 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4583
W20260926 03:37:52.762122 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4583: Not authorized: Client connection negotiation failed: client connection to 127.1.221.130:38777: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:37:52.762354 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4583
I20260926 03:37:52.774211 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4716
W20260926 03:37:52.777092 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4716: Not authorized: Client connection negotiation failed: client connection to 127.1.221.131:36467: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:37:52.777222 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4716
I20260926 03:37:52.789162 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4356
I20260926 03:37:52.790318 4418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:52.892354 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4356
2026-09-26T03:37:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3347 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4861](info): set up 2 sockets
Sep 26 03:37:52 dist-test-slave-lqrk krb5kdc[4861](info): commencing operation
krb5kdc: starting...
W20260926 03:37:53.883735 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:54.059016 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43617
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:54.363611 4874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:54.363937 4874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:54.364059 4874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:54.371155 4874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:54.371279 4874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:54.371353 4874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:54.371451 4874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:54.371512 4874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:54.381240 4874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:54.382529 4874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:54.383858 4874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:54.394938 4879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:54.394953 4880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:54.395131 4882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:54.395778 4874 server_base.cc:1034] running on GCE node
I20260926 03:37:54.396574 4874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:54.397948 4874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:54.399147 4874 hybrid_clock.cc:648] HybridClock initialized: now 1790393874399093 us; error 42 us; skew 500 ppm
I20260926 03:37:54.399515 4874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:54.406185 4874 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:54.407704 4874 webserver.cc:460] Webserver started at http://127.1.221.190:34267/ using document root <none> and password file <none>
I20260926 03:37:54.408411 4874 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:54.408535 4874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:54.408828 4874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:54.411896 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "501032519d6c499a90e6183dfc8e7d6a"
format_stamp: "Formatted at 2026-09-26 03:37:54 on dist-test-slave-lqrk"
server_key: "feba7ae056e3281c5220e40fb532ba6a"
server_key_iv: "4ca910c40c7b7970cf441b9d93e8a83d"
server_key_version: "encryptionkey@0"
I20260926 03:37:54.412686 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "501032519d6c499a90e6183dfc8e7d6a"
format_stamp: "Formatted at 2026-09-26 03:37:54 on dist-test-slave-lqrk"
server_key: "feba7ae056e3281c5220e40fb532ba6a"
server_key_iv: "4ca910c40c7b7970cf441b9d93e8a83d"
server_key_version: "encryptionkey@0"
I20260926 03:37:54.417922 4874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:54.421308 4889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:54.422777 4874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:54.422951 4874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "501032519d6c499a90e6183dfc8e7d6a"
format_stamp: "Formatted at 2026-09-26 03:37:54 on dist-test-slave-lqrk"
server_key: "feba7ae056e3281c5220e40fb532ba6a"
server_key_iv: "4ca910c40c7b7970cf441b9d93e8a83d"
server_key_version: "encryptionkey@0"
I20260926 03:37:54.423137 4874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:54.445622 4874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:54.449707 4874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:54.449973 4874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:54.466615 4874 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43617
I20260926 03:37:54.466638 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43617 every 8 connection(s)
I20260926 03:37:54.468039 4874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:54.471563 4941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:54.474993 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4874
I20260926 03:37:54.475193 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:54.475991 1910 external_mini_cluster.cc:1428] Setting key d49050ca7cc90236780ace259f189040
I20260926 03:37:54.484006 4941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a: Bootstrap starting.
I20260926 03:37:54.487584 4941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:54.488871 4941 log.cc:824] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:54.492010 4941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a: No bootstrap required, opened a new log
I20260926 03:37:54.498379 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:54.477779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.1.221.190:43617)
0926 03:37:54.478529 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:54.478541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:54.478582 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:37:54.480177 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:54.480200 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:54.480613 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:54.480948 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:54.482670 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:54.484036 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:54.485709 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:54.486385 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:54.486509 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:54.491232 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:54.491240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:54.491255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:54.491293 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:54.494307 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:54.495092 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:54.495099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:54.495105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:54.495275 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:54.495674 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:54.495677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:54.495679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:54.496238 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:54.496478 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:54.496890 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 03:37:54.497254 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260926 03:37:54.500679 4941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } }
I20260926 03:37:54.501045 4941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:54.501143 4941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501032519d6c499a90e6183dfc8e7d6a, State: Initialized, Role: FOLLOWER
I20260926 03:37:54.501755 4941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } }
I20260926 03:37:54.501977 4941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:54.502110 4941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:54.502279 4941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:54.502328 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:54.505295 4941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } }
I20260926 03:37:54.505872 4941 leader_election.cc:302] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501032519d6c499a90e6183dfc8e7d6a; no voters:
I20260926 03:37:54.506495 4941 leader_election.cc:288] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:54.506589 4946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:54.507715 4946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [term 1 LEADER]: Becoming Leader. State: Replica: 501032519d6c499a90e6183dfc8e7d6a, State: Running, Role: LEADER
I20260926 03:37:54.508230 4946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } }
I20260926 03:37:54.508622 4941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:54.510818 4947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501032519d6c499a90e6183dfc8e7d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } } }
I20260926 03:37:54.511148 4947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:54.511658 4948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501032519d6c499a90e6183dfc8e7d6a. Latest consensus state: current_term: 1 leader_uuid: "501032519d6c499a90e6183dfc8e7d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501032519d6c499a90e6183dfc8e7d6a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43617 } } }
I20260926 03:37:54.519843 4948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:54.525832 4953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:54.531350 4953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:54.541873 4953 catalog_manager.cc:1347] Generated new cluster ID: fed0ab6c662e4e83bba6f87db0f0c6d3
I20260926 03:37:54.541983 4953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:54.549618 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:54.540796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.1.221.190:43617)
0926 03:37:54.540974 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:54.540980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:54.541563 (+ 583us) server_negotiation.cc:407] Connection header received
0926 03:37:54.541751 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:54.541756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:54.541838 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:54.541951 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:54.543172 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:54.544429 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:54.546208 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:54.546882 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:54.547001 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:54.548029 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:54.548036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:54.548041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:54.548841 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:54.548917 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:54.548951 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:54.549066 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:54.549168 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 03:37:54.549261 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:37:54.564857 4953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:54.566754 4953 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:54.596086 4953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501032519d6c499a90e6183dfc8e7d6a: Generated new TSK 0
I20260926 03:37:54.597301 4953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:54.607015 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4874
I20260926 03:37:54.608250 4936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:54.698616 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4874
2026-09-26T03:37:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1803 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4975](info): set up 2 sockets
Sep 26 03:37:54 dist-test-slave-lqrk krb5kdc[4975](info): commencing operation
krb5kdc: starting...
W20260926 03:37:55.691895 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:55 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:55.876636 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36873
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:56.196762 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:56.197083 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:56.197180 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:56.204615 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:56.204748 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:56.204808 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:56.204871 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:56.204921 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:56.215138 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:56.216511 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:56.217739 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:56.229128 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:56.229152 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:56.229130 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:56.229606 4988 server_base.cc:1034] running on GCE node
I20260926 03:37:56.230623 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:56.232076 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:56.233275 4988 hybrid_clock.cc:648] HybridClock initialized: now 1790393876233214 us; error 45 us; skew 500 ppm
I20260926 03:37:56.233623 4988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:56 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393876, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:56.236577 4988 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:56.237957 4988 webserver.cc:460] Webserver started at http://127.1.221.190:37537/ using document root <none> and password file <none>
I20260926 03:37:56.238613 4988 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:56.238723 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:56.239001 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:56.241964 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "52506b91c21d4d51aa0e9d2521487388"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "57c360b8dc123dd717f1ba57cbc213ab"
server_key_iv: "653fd9a964cb71984783a742698c160f"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.242740 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "52506b91c21d4d51aa0e9d2521487388"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "57c360b8dc123dd717f1ba57cbc213ab"
server_key_iv: "653fd9a964cb71984783a742698c160f"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.247792 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:37:56.250993 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:56.252315 4988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:56.252480 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "52506b91c21d4d51aa0e9d2521487388"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "57c360b8dc123dd717f1ba57cbc213ab"
server_key_iv: "653fd9a964cb71984783a742698c160f"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.252646 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:56.273501 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:56.277251 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:56.277515 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:56.294013 4988 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36873
I20260926 03:37:56.294034 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36873 every 8 connection(s)
I20260926 03:37:56.295308 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:56.298959 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:56.303972 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 4988
I20260926 03:37:56.304211 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:56.304989 1910 external_mini_cluster.cc:1428] Setting key 7de94a92f63817fd3ddb907de1e83981
I20260926 03:37:56.310253 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388: Bootstrap starting.
I20260926 03:37:56.313788 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:56.315018 5055 log.cc:824] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:56.318385 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388: No bootstrap required, opened a new log
Sep 26 03:37:56 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:56.327057 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } }
I20260926 03:37:56.327422 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:56.327481 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:56.307041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.1.221.190:36873)
0926 03:37:56.307878 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:56.307891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:56.307937 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:37:56.309669 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:56.309692 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:56.310089 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:56.310439 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:56.312069 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.313337 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.315039 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.315940 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.316123 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:56.320486 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:56.320494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:56.320509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:56.320549 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:56.323474 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:56.324254 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:56.324260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:56.324266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:56.324438 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:56.324811 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:56.324814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:56.324816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:56.325340 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:56.325566 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:56.325997 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 03:37:56.326351 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":392,"threads_started":1}
I20260926 03:37:56.327533 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52506b91c21d4d51aa0e9d2521487388, State: Initialized, Role: FOLLOWER
I20260926 03:37:56.328069 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } }
I20260926 03:37:56.328289 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:56.328408 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:56.328578 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:56.330754 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } }
I20260926 03:37:56.331199 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52506b91c21d4d51aa0e9d2521487388; no voters:
I20260926 03:37:56.331764 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:56.331874 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:56.332458 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [term 1 LEADER]: Becoming Leader. State: Replica: 52506b91c21d4d51aa0e9d2521487388, State: Running, Role: LEADER
I20260926 03:37:56.333425 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } }
I20260926 03:37:56.333756 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:56.336618 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52506b91c21d4d51aa0e9d2521487388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } } }
I20260926 03:37:56.336381 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52506b91c21d4d51aa0e9d2521487388. Latest consensus state: current_term: 1 leader_uuid: "52506b91c21d4d51aa0e9d2521487388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52506b91c21d4d51aa0e9d2521487388" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36873 } } }
I20260926 03:37:56.337258 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:56.337446 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:56.340093 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:56.344274 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:56.353780 5069 catalog_manager.cc:1347] Generated new cluster ID: a8799b3e20cc4334b365d0709f85c3a8
I20260926 03:37:56.353889 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:56.361683 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:56.363546 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:56.376852 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52506b91c21d4d51aa0e9d2521487388: Generated new TSK 0
I20260926 03:37:56.377725 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:56.432157 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:56.741861 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:56.742161 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:56.742301 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:56.749811 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:56.749939 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:56.750094 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:37:56.760226 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:56.761531 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:56.762827 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:56.775198 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:56.775172 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:56.775466 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:56.776065 5081 server_base.cc:1034] running on GCE node
I20260926 03:37:56.776716 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:56.777665 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:56.778877 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790393876778816 us; error 54 us; skew 500 ppm
I20260926 03:37:56.779183 5081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:56 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393876, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:56.782289 5081 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:37:56.783797 5081 webserver.cc:460] Webserver started at http://127.1.221.129:43905/ using document root <none> and password file <none>
I20260926 03:37:56.784472 5081 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:56.784569 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:56.784826 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:56.787662 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "a55381c642934a419d0efeac3872b536"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "59043a78f11a8ddee18a874a95c259b6"
server_key_iv: "96118555570bc71d9cad4e27068ffc7e"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.788384 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "a55381c642934a419d0efeac3872b536"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "59043a78f11a8ddee18a874a95c259b6"
server_key_iv: "96118555570bc71d9cad4e27068ffc7e"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.793246 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:56.796566 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:56.797901 5081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:56.798036 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "a55381c642934a419d0efeac3872b536"
format_stamp: "Formatted at 2026-09-26 03:37:56 on dist-test-slave-lqrk"
server_key: "59043a78f11a8ddee18a874a95c259b6"
server_key_iv: "96118555570bc71d9cad4e27068ffc7e"
server_key_version: "encryptionkey@0"
I20260926 03:37:56.798194 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:56.830587 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:56.834934 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:56.835198 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:56.836161 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:56.837440 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:56.837515 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:56.837644 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:56.837711 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:56.865937 5081 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:34169
I20260926 03:37:56.865988 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:34169 every 8 connection(s)
I20260926 03:37:56.867213 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:37:56.871587 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5081
I20260926 03:37:56.871764 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:37:56.872380 1910 external_mini_cluster.cc:1428] Setting key 732e1052db30a7f4cba0ad60bfe8739c
Sep 26 03:37:56 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393876, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:56.886735 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:56.869693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58977 (local address 127.1.221.190:36873)
0926 03:37:56.870113 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:56.870118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:56.871045 (+ 927us) server_negotiation.cc:407] Connection header received
0926 03:37:56.872915 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:56.872921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:56.873002 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:56.873096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:56.874830 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.875620 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.877277 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.877663 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.877766 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:56.881328 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:56.881333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:56.881337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:56.881376 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:56.883745 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:56.884506 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:56.884512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:56.884516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:56.884654 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:56.885058 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:56.885063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:56.885067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:56.885337 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:56.885469 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:56.886153 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 03:37:56.886380 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:37:56.887792 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:56.870208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36873 (local address 127.1.221.129:58977)
0926 03:37:56.870884 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:56.870929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:56.872641 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:56.873387 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:56.873399 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:56.874093 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:56.874675 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:56.874692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.875783 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.875787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:56.877147 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:56.877160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:56.877874 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:56.877882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:56.878097 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:56.878706 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:56.878733 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:56.881100 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:56.883893 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:56.883903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:56.883922 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:56.884331 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:56.884762 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:56.884765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:56.884768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:56.884955 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:56.885459 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:56.885467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:56.885806 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 03:37:56.886064 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 03:37:56.886625 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
I20260926 03:37:56.889452 5209 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36873
I20260926 03:37:56.889755 5209 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:37:56.890597 5209 heartbeater.cc:511] Master 127.1.221.190:36873 requested a full tablet report, sending...
I20260926 03:37:56.893083 5020 ts_manager.cc:194] Registered new tserver with Master: a55381c642934a419d0efeac3872b536 (127.1.221.129:34169)
I20260926 03:37:56.895035 5020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:58977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:37:56.914867 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:57.225023 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:57.225324 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:57.225454 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:57.232813 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:57.232939 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:57.233083 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:37:57.242964 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:57.244344 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:57.245679 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:57.258252 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:57.258204 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:57.258275 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:57.258996 5215 server_base.cc:1034] running on GCE node
I20260926 03:37:57.259610 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:57.260529 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:57.261718 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790393877261673 us; error 35 us; skew 500 ppm
I20260926 03:37:57.261987 5215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:57.264845 5215 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:37:57.266222 5215 webserver.cc:460] Webserver started at http://127.1.221.130:42845/ using document root <none> and password file <none>
I20260926 03:37:57.266850 5215 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:57.266949 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:57.267194 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:57.270025 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "5ed710034ab84af2b012c58ebdd2aa2b"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "6263df8a1c38d07b0a9cde944c04c2f1"
server_key_iv: "9a4db77e51d6f85249058f8de6a69943"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.270738 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "5ed710034ab84af2b012c58ebdd2aa2b"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "6263df8a1c38d07b0a9cde944c04c2f1"
server_key_iv: "9a4db77e51d6f85249058f8de6a69943"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.275676 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:37:57.279155 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.280555 5215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:57.280697 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "5ed710034ab84af2b012c58ebdd2aa2b"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "6263df8a1c38d07b0a9cde944c04c2f1"
server_key_iv: "9a4db77e51d6f85249058f8de6a69943"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.280864 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:57.311506 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:57.316656 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:57.316910 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:57.317859 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:57.319168 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:57.319238 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.319325 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:57.319365 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.349081 5215 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:33401
I20260926 03:37:57.349149 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:33401 every 8 connection(s)
I20260926 03:37:57.350309 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:37:57.351640 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5215
I20260926 03:37:57.351799 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:37:57.352308 1910 external_mini_cluster.cc:1428] Setting key 4849f5a03612fa5120b6f4be662ee8db
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:37:57.372064 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.353014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:37275 (local address 127.1.221.190:36873)
0926 03:37:57.353191 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.353197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.354781 (+ 1584us) server_negotiation.cc:407] Connection header received
0926 03:37:57.357222 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.357226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.357293 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.357414 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.359314 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.360107 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.361778 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.362175 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.362286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.365926 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.365934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.365939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.365980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.369200 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.369886 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.369892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.369895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.370062 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.370455 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.370461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.370464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.370817 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.371007 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.371632 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.371828 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:37:57.373059 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.353729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36873 (local address 127.1.221.130:37275)
0926 03:37:57.354606 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:57.354656 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:57.356935 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:57.357657 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:57.357676 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:57.358365 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:57.359143 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:57.359169 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.360264 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.360269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:57.361613 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:57.361626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.362400 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.362408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:57.362633 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.363276 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:57.363300 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:57.365661 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:57.369328 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:57.369336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:57.369350 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:57.369773 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:57.370160 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:57.370163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:57.370165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:57.370349 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:57.370948 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:57.370954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:57.371275 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 03:37:57.371545 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 03:37:57.371975 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":328,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:37:57.374567 5343 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36873
I20260926 03:37:57.374876 5343 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:57.375595 5343 heartbeater.cc:511] Master 127.1.221.190:36873 requested a full tablet report, sending...
I20260926 03:37:57.377032 5020 ts_manager.cc:194] Registered new tserver with Master: 5ed710034ab84af2b012c58ebdd2aa2b (127.1.221.130:33401)
I20260926 03:37:57.378021 5020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:37275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:37:57.394352 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:57.705282 5348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:57.705581 5348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:57.705674 5348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:57.712949 5348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:57.713069 5348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:57.713188 5348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:37:57.723027 5348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:57.724334 5348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:57.725667 5348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:57.738723 5353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:57.738688 5356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:57.738754 5354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:57.739600 5348 server_base.cc:1034] running on GCE node
I20260926 03:37:57.740298 5348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:57.741268 5348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:57.742477 5348 hybrid_clock.cc:648] HybridClock initialized: now 1790393877742423 us; error 43 us; skew 500 ppm
I20260926 03:37:57.742786 5348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:57.745930 5348 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:37:57.747325 5348 webserver.cc:460] Webserver started at http://127.1.221.131:40755/ using document root <none> and password file <none>
I20260926 03:37:57.748027 5348 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:57.748155 5348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:57.748443 5348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:57.751266 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "ce5282709ff0469eb40ccd5fb99742fb"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "65d1b19ff798a43042f1b71289f69032"
server_key_iv: "02de10f6a56c1b7c9e5008a45133999a"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.752059 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "ce5282709ff0469eb40ccd5fb99742fb"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "65d1b19ff798a43042f1b71289f69032"
server_key_iv: "02de10f6a56c1b7c9e5008a45133999a"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.757088 5348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:37:57.760457 5363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.761797 5348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:37:57.761950 5348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "ce5282709ff0469eb40ccd5fb99742fb"
format_stamp: "Formatted at 2026-09-26 03:37:57 on dist-test-slave-lqrk"
server_key: "65d1b19ff798a43042f1b71289f69032"
server_key_iv: "02de10f6a56c1b7c9e5008a45133999a"
server_key_version: "encryptionkey@0"
I20260926 03:37:57.762118 5348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:57.783625 5348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:57.787536 5348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:57.787801 5348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:57.788807 5348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:37:57.790083 5348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:37:57.790161 5348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.790279 5348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:37:57.790331 5348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:57.819729 5348 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36343
I20260926 03:37:57.819785 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36343 every 8 connection(s)
I20260926 03:37:57.820988 5348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:37:57.821569 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5348
I20260926 03:37:57.821731 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:37:57.822306 1910 external_mini_cluster.cc:1428] Setting key 4ffb9bb5ddb28e1a68db9d38a3dcba18
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:57.840749 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.823518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:42255 (local address 127.1.221.190:36873)
0926 03:37:57.823690 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.823696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.825238 (+ 1542us) server_negotiation.cc:407] Connection header received
0926 03:37:57.827007 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.827012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.827093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.827182 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.828650 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.829389 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.830858 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.831276 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.831404 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.834772 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.834777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.834781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.834817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.837824 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.838555 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.838559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.838561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.838684 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.839067 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.839070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.839072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.839449 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.839559 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.840274 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.840508 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:37:57.841714 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.824213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36873 (local address 127.1.221.131:42255)
0926 03:37:57.825080 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:57.825123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:57.826753 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:57.827445 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:57.827453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:57.827939 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:57.828484 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:57.828511 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.829522 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.829525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:57.830725 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:57.830737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.831509 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.831517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:57.831746 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.832369 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:57.832392 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:57.834560 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:57.837963 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:57.837971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:57.837984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:57.838456 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:57.838763 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:57.838766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:57.838768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:57.838982 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:57.839568 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:57.839577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:57.839953 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 03:37:57.840192 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 03:37:57.840658 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":370,"threads_started":1}
I20260926 03:37:57.843070 5476 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36873
I20260926 03:37:57.843415 5476 heartbeater.cc:464] Registering TS with master...
I20260926 03:37:57.844142 5476 heartbeater.cc:511] Master 127.1.221.190:36873 requested a full tablet report, sending...
I20260926 03:37:57.845636 5020 ts_manager.cc:194] Registered new tserver with Master: ce5282709ff0469eb40ccd5fb99742fb (127.1.221.131:36343)
I20260926 03:37:57.846525 5020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:42255
I20260926 03:37:57.850026 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:57.897120 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.883262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.1.221.190:36873)
0926 03:37:57.883502 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.883510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.883934 (+ 424us) server_negotiation.cc:407] Connection header received
0926 03:37:57.884159 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.884165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.884246 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.884341 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.885761 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.886506 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.887753 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.888154 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.888265 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.891877 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.891882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.891885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.891924 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.894481 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.895132 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.895137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.895139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.895278 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.895709 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.895713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.895715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.896078 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.896303 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.896607 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.896826 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:37:57.897661 5020 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48566
I20260926 03:37:57.898195 5209 heartbeater.cc:503] Master 127.1.221.190:36873 was elected leader, sending a full tablet report...
I20260926 03:37:57.899236 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5081
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:37:57.913919 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.900240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.1.221.129:34169)
0926 03:37:57.900810 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.900816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.900847 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:37:57.900898 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.900903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.901111 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.901230 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.902391 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.903488 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.904708 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.905252 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.905350 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.908906 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.908917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.908932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.908964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.911609 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.912141 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.912147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.912150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.912257 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.912613 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.912616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.912618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.912946 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.913156 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.913355 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.913604 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
W20260926 03:37:57.915227 5204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57722
W20260926 03:37:57.916306 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5081: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:57.916508 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5081
I20260926 03:37:57.928798 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5215
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:37:57.944060 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.929636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.1.221.130:33401)
0926 03:37:57.930112 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.930119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.930144 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:37:57.930204 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.930210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.930433 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.930556 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.931740 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.932724 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.933967 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.934533 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.934644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.938469 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.938482 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.938496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.938543 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.941398 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.941979 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.941987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.941989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.942131 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.942462 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.942465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.942467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.942816 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.943060 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.943538 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.943756 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":246,"threads_started":1}
W20260926 03:37:57.945250 5338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45558
W20260926 03:37:57.946290 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:57.946425 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5215
I20260926 03:37:57.958794 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5348
Sep 26 03:37:57 dist-test-slave-lqrk krb5kdc[4975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:37:57.974375 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:57.959761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.1.221.131:36343)
0926 03:37:57.960396 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:57.960404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:57.960431 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:37:57.960488 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:57.960495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:57.960706 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:57.960860 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:57.962126 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.963160 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.964453 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:57.965012 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:57.965124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:57.969267 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:57.969278 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:57.969292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:57.969332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:57.971927 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.972559 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.972565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.972567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.972683 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:57.972994 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:57.972999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:57.973001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:57.973379 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:57.973608 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:57.973886 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:37:57.974098 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":328,"threads_started":1}
W20260926 03:37:57.975648 5471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49002
W20260926 03:37:57.976758 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:57.976903 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5348
I20260926 03:37:57.988529 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 4988
I20260926 03:37:57.989715 5050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:58.090579 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 4988
2026-09-26T03:37:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3394 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:58 dist-test-slave-lqrk krb5kdc[5493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:58 dist-test-slave-lqrk krb5kdc[5493](info): set up 2 sockets
Sep 26 03:37:58 dist-test-slave-lqrk krb5kdc[5493](info): commencing operation
krb5kdc: starting...
W20260926 03:37:59.087687 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:37:59 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:37:59.271097 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41225
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:59.578122 5506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:59.578439 5506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:59.578574 5506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:59.585800 5506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:59.585925 5506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:59.586004 5506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:59.586078 5506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:59.586138 5506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:59.596432 5506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:59.597723 5506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:59.599022 5506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:59.610731 5511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:59.610692 5512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:59.610780 5514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:59.611744 5506 server_base.cc:1034] running on GCE node
I20260926 03:37:59.612618 5506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:59.614171 5506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:59.615371 5506 hybrid_clock.cc:648] HybridClock initialized: now 1790393879615324 us; error 46 us; skew 500 ppm
I20260926 03:37:59.615739 5506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:59 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393879, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:59.621598 5506 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:37:59.623099 5506 webserver.cc:460] Webserver started at http://127.1.221.190:42033/ using document root <none> and password file <none>
I20260926 03:37:59.623843 5506 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:59.623962 5506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:59.624261 5506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:59.627187 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "5838ed865ebd448f915858284fafa108"
format_stamp: "Formatted at 2026-09-26 03:37:59 on dist-test-slave-lqrk"
server_key: "f71fa0e2f4083c3486420b1f15336431"
server_key_iv: "111130a910d43cb6237ae744573f06f2"
server_key_version: "encryptionkey@0"
I20260926 03:37:59.627997 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "5838ed865ebd448f915858284fafa108"
format_stamp: "Formatted at 2026-09-26 03:37:59 on dist-test-slave-lqrk"
server_key: "f71fa0e2f4083c3486420b1f15336431"
server_key_iv: "111130a910d43cb6237ae744573f06f2"
server_key_version: "encryptionkey@0"
I20260926 03:37:59.633145 5506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:37:59.636631 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:59.637998 5506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:59.638168 5506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "5838ed865ebd448f915858284fafa108"
format_stamp: "Formatted at 2026-09-26 03:37:59 on dist-test-slave-lqrk"
server_key: "f71fa0e2f4083c3486420b1f15336431"
server_key_iv: "111130a910d43cb6237ae744573f06f2"
server_key_version: "encryptionkey@0"
I20260926 03:37:59.638360 5506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:59.649324 5506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:59.653023 5506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:59.653280 5506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:59.669891 5506 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41225
I20260926 03:37:59.669955 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41225 every 8 connection(s)
I20260926 03:37:59.672466 5506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:59.676478 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5506
I20260926 03:37:59.676729 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:37:59.677480 1910 external_mini_cluster.cc:1428] Setting key dd358ac8de22161eac6821353f194e1b
I20260926 03:37:59.677809 5573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:59.695273 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108: Bootstrap starting.
Sep 26 03:37:59 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:37:59.699607 5573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:59.700995 5573 log.cc:824] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:59.704854 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:59.680409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.1.221.190:41225)
0926 03:37:59.681605 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:59.681625 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:59.681674 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:37:59.683943 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:59.683978 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:59.684536 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:59.685173 (+ 637us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:59.686525 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:59.688548 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:59.690314 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:59.691451 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:59.691627 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:59.696617 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:59.696628 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:59.696649 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:59.696705 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:59.700619 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:59.701319 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:59.701328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:59.701337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:59.701534 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:59.701908 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:59.701913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:59.701916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:59.702556 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:59.702840 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:59.703265 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 03:37:59.703715 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":723,"threads_started":1}
I20260926 03:37:59.708559 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108: No bootstrap required, opened a new log
I20260926 03:37:59.719661 5573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } }
I20260926 03:37:59.720238 5573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:59.720330 5573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5838ed865ebd448f915858284fafa108, State: Initialized, Role: FOLLOWER
I20260926 03:37:59.720957 5573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } }
I20260926 03:37:59.721202 5573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:59.721302 5573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:59.721434 5573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:59.723593 5573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } }
I20260926 03:37:59.724083 5573 leader_election.cc:302] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5838ed865ebd448f915858284fafa108; no voters:
I20260926 03:37:59.724640 5573 leader_election.cc:288] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:59.724777 5578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:59.725399 5578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [term 1 LEADER]: Becoming Leader. State: Replica: 5838ed865ebd448f915858284fafa108, State: Running, Role: LEADER
I20260926 03:37:59.725907 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } }
I20260926 03:37:59.726687 5573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:59.728716 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5838ed865ebd448f915858284fafa108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } } }
I20260926 03:37:59.728782 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5838ed865ebd448f915858284fafa108. Latest consensus state: current_term: 1 leader_uuid: "5838ed865ebd448f915858284fafa108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5838ed865ebd448f915858284fafa108" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41225 } } }
I20260926 03:37:59.729336 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:59.729336 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:59.737082 5585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:59.741812 5585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:59.750695 5585 catalog_manager.cc:1347] Generated new cluster ID: 7c0ea1abf17145bfa63f31a4f7239651
I20260926 03:37:59.750820 5585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:59.771183 5585 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:59.772588 5585 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:59.780194 5585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5838ed865ebd448f915858284fafa108: Generated new TSK 0
I20260926 03:37:59.781180 5585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:37:59.832913 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:00.150734 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:00.151024 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:00.151118 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:00.159504 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:00.159626 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:00.159746 5599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:00.169838 5599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:00.171132 5599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:00.172511 5599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:00.185487 5604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:00.185487 5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:00.185461 5605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:00.186303 5599 server_base.cc:1034] running on GCE node
I20260926 03:38:00.186926 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:00.187930 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:00.189119 5599 hybrid_clock.cc:648] HybridClock initialized: now 1790393880189065 us; error 45 us; skew 500 ppm
I20260926 03:38:00.189424 5599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:00 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393880, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:00.192476 5599 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:38:00.193820 5599 webserver.cc:460] Webserver started at http://127.1.221.129:42255/ using document root <none> and password file <none>
I20260926 03:38:00.194518 5599 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:00.194622 5599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:00.194875 5599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:00.197687 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "235ce788dd82452fb447afc0dcc59f4f"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "acf49a95a2731a75e70e3c60d484a9d9"
server_key_iv: "e0c3764f9f1006897330c2e9b9772219"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.198402 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "235ce788dd82452fb447afc0dcc59f4f"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "acf49a95a2731a75e70e3c60d484a9d9"
server_key_iv: "e0c3764f9f1006897330c2e9b9772219"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.203439 5599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:38:00.206799 5614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.208176 5599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:38:00.208314 5599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "235ce788dd82452fb447afc0dcc59f4f"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "acf49a95a2731a75e70e3c60d484a9d9"
server_key_iv: "e0c3764f9f1006897330c2e9b9772219"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.208483 5599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:00.250558 5599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:00.254586 5599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:00.254844 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:00.255807 5599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:00.257118 5599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:00.257189 5599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.257282 5599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:00.257323 5599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.286295 5599 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36275
I20260926 03:38:00.286331 5726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36275 every 8 connection(s)
I20260926 03:38:00.287547 5599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:00.290722 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5599
I20260926 03:38:00.290922 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:00.291742 1910 external_mini_cluster.cc:1428] Setting key 86deb0bf8859305fcd24164afeae83f3
Sep 26 03:38:00 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393880, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:00.309166 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:00.290007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38707 (local address 127.1.221.190:41225)
0926 03:38:00.290473 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:00.290479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:00.291574 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 03:38:00.293759 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:00.293764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:00.293826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:00.293914 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:00.296004 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.296725 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.298826 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.299209 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.299311 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:00.302777 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:00.302784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:00.302789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:00.302827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:00.305774 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:00.306561 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:00.306569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:00.306572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:00.306776 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:00.307413 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:00.307419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:00.307423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:00.307704 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:00.307843 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:00.308585 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 03:38:00.308815 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:38:00.310191 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:00.290564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41225 (local address 127.1.221.129:38707)
0926 03:38:00.291354 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:00.291446 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:00.293481 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:00.294201 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:00.294213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:00.294952 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:00.295841 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:00.295868 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.296890 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.296895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:00.298693 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:00.298708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.299461 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.299470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:00.299708 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:00.300364 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:00.300387 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:00.302513 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:00.305912 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:00.305919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:00.305936 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:00.306377 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:00.306921 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:00.306927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:00.306931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:00.307205 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:00.307843 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:00.307852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:00.308200 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 03:38:00.308444 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 03:38:00.309022 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":309,"threads_started":1}
I20260926 03:38:00.311751 5727 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41225
I20260926 03:38:00.312068 5727 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:38:00.312876 5727 heartbeater.cc:511] Master 127.1.221.190:41225 requested a full tablet report, sending...
I20260926 03:38:00.315049 5538 ts_manager.cc:194] Registered new tserver with Master: 235ce788dd82452fb447afc0dcc59f4f (127.1.221.129:36275)
I20260926 03:38:00.316517 5538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:38707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:38:00.333783 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:00.645910 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:00.646201 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:00.646337 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:00.653656 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:00.653766 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:00.653909 5733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:00.663851 5733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:00.665117 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:00.666419 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:00.679194 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:00.679253 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:00.679847 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:00.680649 5733 server_base.cc:1034] running on GCE node
I20260926 03:38:00.681257 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:00.682240 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:00.683470 5733 hybrid_clock.cc:648] HybridClock initialized: now 1790393880683403 us; error 57 us; skew 500 ppm
I20260926 03:38:00.683792 5733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:00 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393880, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:00.686776 5733 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:38:00.688263 5733 webserver.cc:460] Webserver started at http://127.1.221.130:39117/ using document root <none> and password file <none>
I20260926 03:38:00.688937 5733 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:00.689044 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:00.689368 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:00.692389 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "93789296865a46cfa5e583a0e0b7f2e7"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "4c4c085f60b2f275af0753ab4107a389"
server_key_iv: "ad9b7ee20cafc6e5995ad35db8191453"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.693127 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "93789296865a46cfa5e583a0e0b7f2e7"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "4c4c085f60b2f275af0753ab4107a389"
server_key_iv: "ad9b7ee20cafc6e5995ad35db8191453"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.698174 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:00.701623 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.703018 5733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:00.703169 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "93789296865a46cfa5e583a0e0b7f2e7"
format_stamp: "Formatted at 2026-09-26 03:38:00 on dist-test-slave-lqrk"
server_key: "4c4c085f60b2f275af0753ab4107a389"
server_key_iv: "ad9b7ee20cafc6e5995ad35db8191453"
server_key_version: "encryptionkey@0"
I20260926 03:38:00.703362 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:00.720471 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:00.724364 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:00.724647 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:00.725574 5733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:00.726990 5733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:00.727067 5733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.727191 5733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:00.727249 5733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:00.757407 5733 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:34371
I20260926 03:38:00.757452 5860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:34371 every 8 connection(s)
I20260926 03:38:00.758723 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:00.760941 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5733
I20260926 03:38:00.761111 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:00.761584 1910 external_mini_cluster.cc:1428] Setting key 666622754a98d85f852d79816b2d89a3
Sep 26 03:38:00 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393880, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:00.779094 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:00.761175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:44199 (local address 127.1.221.190:41225)
0926 03:38:00.761389 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:00.761395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:00.762643 (+ 1248us) server_negotiation.cc:407] Connection header received
0926 03:38:00.764510 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:00.764515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:00.764602 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:00.764707 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:00.766163 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.767111 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.768759 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.769169 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.769275 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:00.773365 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:00.773370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:00.773374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:00.773408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:00.775893 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:00.776811 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:00.776814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:00.776816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:00.776934 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:00.777418 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:00.777421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:00.777423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:00.777693 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:00.777820 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:00.778678 (+ 858us) server_negotiation.cc:299] Negotiation successful
0926 03:38:00.778858 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:38:00.780604 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:00.761675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41225 (local address 127.1.221.130:44199)
0926 03:38:00.762476 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:00.762524 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:00.764240 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:00.764913 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:00.764921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:00.765431 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:00.766003 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:00.766020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.767251 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.767255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:00.768582 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:00.768598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:00.769409 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:00.769417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:00.769739 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:00.770455 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:00.770485 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:00.773094 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:00.776041 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:00.776051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:00.776071 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:00.776699 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:00.777033 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:00.777037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:00.777041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:00.777308 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:00.777810 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:00.777818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:00.778286 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 03:38:00.778616 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 03:38:00.779191 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":306,"threads_started":1}
I20260926 03:38:00.782027 5861 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41225
I20260926 03:38:00.782310 5861 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:38:00.783206 5861 heartbeater.cc:511] Master 127.1.221.190:41225 requested a full tablet report, sending...
I20260926 03:38:00.784832 5538 ts_manager.cc:194] Registered new tserver with Master: 93789296865a46cfa5e583a0e0b7f2e7 (127.1.221.130:34371)
I20260926 03:38:00.785614 5538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:44199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:38:00.805011 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:01.119537 5866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:01.119864 5866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:01.119968 5866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:01.127314 5866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:01.127449 5866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:01.127604 5866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:01.138813 5866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:01.140348 5866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:01.141779 5866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:01.154935 5874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:01.155272 5872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:01.155153 5871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:01.158185 5866 server_base.cc:1034] running on GCE node
I20260926 03:38:01.158831 5866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:01.159955 5866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:01.161165 5866 hybrid_clock.cc:648] HybridClock initialized: now 1790393881161099 us; error 55 us; skew 500 ppm
I20260926 03:38:01.161487 5866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:01.164907 5866 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:38:01.166438 5866 webserver.cc:460] Webserver started at http://127.1.221.131:41529/ using document root <none> and password file <none>
I20260926 03:38:01.167163 5866 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:01.167301 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:01.167627 5866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:01.171090 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "3bd7e6a1d7704f74b98290db491c3056"
format_stamp: "Formatted at 2026-09-26 03:38:01 on dist-test-slave-lqrk"
server_key: "c52efcf7ff10d1822c4b26769224ef35"
server_key_iv: "83a99791724a7bd126ce93080b26c638"
server_key_version: "encryptionkey@0"
I20260926 03:38:01.171919 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "3bd7e6a1d7704f74b98290db491c3056"
format_stamp: "Formatted at 2026-09-26 03:38:01 on dist-test-slave-lqrk"
server_key: "c52efcf7ff10d1822c4b26769224ef35"
server_key_iv: "83a99791724a7bd126ce93080b26c638"
server_key_version: "encryptionkey@0"
I20260926 03:38:01.177657 5866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:38:01.181452 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:01.182876 5866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:01.183030 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "3bd7e6a1d7704f74b98290db491c3056"
format_stamp: "Formatted at 2026-09-26 03:38:01 on dist-test-slave-lqrk"
server_key: "c52efcf7ff10d1822c4b26769224ef35"
server_key_iv: "83a99791724a7bd126ce93080b26c638"
server_key_version: "encryptionkey@0"
I20260926 03:38:01.183483 5866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:01.204321 5866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:01.208819 5866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:01.209086 5866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:01.210034 5866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:01.211463 5866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:01.211541 5866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:01.211694 5866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:01.211756 5866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:01.242276 5866 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43971
I20260926 03:38:01.242308 5993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43971 every 8 connection(s)
I20260926 03:38:01.243628 5866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:01.252831 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 5866
I20260926 03:38:01.253038 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:01.253528 1910 external_mini_cluster.cc:1428] Setting key ef04d6ddd53afba806610c5cb80ec51f
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:01.263345 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.246067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55561 (local address 127.1.221.190:41225)
0926 03:38:01.246261 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.246266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.247427 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 03:38:01.249189 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.249194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.249295 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.249380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.250789 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.251567 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.253337 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.253746 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.253839 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.257617 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.257622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.257625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.257659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.260389 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.261100 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.261106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.261110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.261241 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.261657 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.261662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.261666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.261942 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.262066 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.262773 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.262987 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:38:01.264362 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.246549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41225 (local address 127.1.221.131:55561)
0926 03:38:01.247240 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:01.247279 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:01.248942 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:01.249581 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:01.249590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:01.250090 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:01.250649 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:01.250668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.251680 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.251683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:01.253209 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:01.253224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.253977 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.253985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:01.254195 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.254851 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:01.254879 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:01.257428 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:01.260527 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:01.260537 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:01.260556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:01.260985 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:01.261349 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:01.261354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:01.261358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:01.261554 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:01.262062 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:01.262070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:01.262421 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 03:38:01.262667 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 03:38:01.263215 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260926 03:38:01.265776 5994 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41225
I20260926 03:38:01.266045 5994 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:01.266789 5994 heartbeater.cc:511] Master 127.1.221.190:41225 requested a full tablet report, sending...
I20260926 03:38:01.268424 5538 ts_manager.cc:194] Registered new tserver with Master: 3bd7e6a1d7704f74b98290db491c3056 (127.1.221.131:43971)
I20260926 03:38:01.269145 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:38:01.269637 5538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:55561
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:01.319406 5727 heartbeater.cc:503] Master 127.1.221.190:41225 was elected leader, sending a full tablet report...
I20260926 03:38:01.325344 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.309004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.1.221.129:36275)
0926 03:38:01.309597 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.309605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.309632 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:01.309691 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.309698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.309927 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.310062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.311428 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.313081 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.314273 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.315158 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.315295 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.319186 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.319203 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.319224 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.319281 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.322610 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.323263 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.323273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.323277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.323465 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.323819 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.323824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.323827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.324165 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.324443 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.324750 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.325002 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":290,"threads_started":1}
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:01.342182 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.329452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.1.221.190:41225)
0926 03:38:01.329645 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.329651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.329856 (+ 205us) server_negotiation.cc:407] Connection header received
0926 03:38:01.330013 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.330017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.330077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.330154 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.331358 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.332126 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.333317 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.333720 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.333837 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.337468 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.337473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.337477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.337510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.339773 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.340367 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.340371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.340373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.340499 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.340816 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.340822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.340825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.341138 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.341366 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.341673 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.341887 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 03:38:01.342720 5538 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610
I20260926 03:38:01.343703 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5599
I20260926 03:38:01.355688 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.344700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.1.221.129:36275)
0926 03:38:01.344894 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.344900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.345013 (+ 113us) server_negotiation.cc:407] Connection header received
0926 03:38:01.345170 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.345176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.345254 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.345332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.346463 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.347135 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.348371 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.348807 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.348936 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.351000 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.351005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.351009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.351041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.353284 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.353853 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.353858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.353863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.353982 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.354307 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.354312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.354316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.354562 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.354786 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.355069 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.355278 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:38:01.356287 5722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:01.437260 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5599
I20260926 03:38:01.450228 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5733
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:38:01.465885 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.451110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.1.221.130:34371)
0926 03:38:01.451682 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.451689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.451714 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:38:01.451772 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.451779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.451993 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.452122 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.453203 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.454204 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.455352 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.455977 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.456071 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.460091 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.460106 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.460122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.460162 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.462979 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.463765 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.463774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.463776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.463908 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.464285 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.464290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.464292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.464653 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.464948 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.465279 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.465552 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":329,"threads_started":1}
I20260926 03:38:01.467135 5856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:01.548028 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5733
I20260926 03:38:01.560364 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5866
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[5493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:38:01.575940 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:01.561446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.1.221.131:43971)
0926 03:38:01.562098 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:01.562105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:01.562134 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:38:01.562193 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:01.562200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:01.562422 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:01.562551 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:01.563828 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.564934 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.566144 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:01.566691 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:01.566798 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:01.570829 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:01.570841 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:01.570857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:01.570894 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:01.573453 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.574042 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.574049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.574051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.574164 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:01.574485 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:01.574490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:01.574492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:01.574888 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:01.575172 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:01.575326 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 03:38:01.575614 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":351,"threads_started":1}
I20260926 03:38:01.577195 5989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:01.659240 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5866
I20260926 03:38:01.671860 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 5506
I20260926 03:38:01.673064 5568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:01.774187 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 5506
2026-09-26T03:38:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3684 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[6013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[6013](info): set up 2 sockets
Sep 26 03:38:01 dist-test-slave-lqrk krb5kdc[6013](info): commencing operation
krb5kdc: starting...
W20260926 03:38:02.771672 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:02 dist-test-slave-lqrk krb5kdc[6013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:38:02.959510 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40783
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:03.272236 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:03.272543 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:03.272643 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:03.279917 6026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:03.280035 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:03.280088 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:03.280143 6026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:03.280189 6026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:03.290109 6026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:03.291344 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:03.292618 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:03.303970 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:03.303907 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:03.303934 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:03.305011 6026 server_base.cc:1034] running on GCE node
I20260926 03:38:03.305825 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:03.307286 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:03.308488 6026 hybrid_clock.cc:648] HybridClock initialized: now 1790393883308420 us; error 62 us; skew 500 ppm
I20260926 03:38:03.308841 6026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:03 dist-test-slave-lqrk krb5kdc[6013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393883, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:03.311843 6026 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:38:03.313050 6026 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:03.313153 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:03.313452 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:03.316391 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "9a0f8574a3844d3c84bbb09d71aafb7f"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "9c0c3d51ac2bac5ed78f865e344bed44"
server_key_iv: "b6ab0e5c15612a53ec1c239345f72e01"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.317142 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "9a0f8574a3844d3c84bbb09d71aafb7f"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "9c0c3d51ac2bac5ed78f865e344bed44"
server_key_iv: "b6ab0e5c15612a53ec1c239345f72e01"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.322085 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:38:03.325742 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:03.327040 6026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:03.327199 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "9a0f8574a3844d3c84bbb09d71aafb7f"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "9c0c3d51ac2bac5ed78f865e344bed44"
server_key_iv: "b6ab0e5c15612a53ec1c239345f72e01"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.327420 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:03.339123 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:03.342898 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:03.343132 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:03.359905 6026 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40783
I20260926 03:38:03.359879 6091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40783 every 8 connection(s)
I20260926 03:38:03.360817 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:03.364475 6092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:03.366621 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6026
I20260926 03:38:03.366837 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:03.367449 1910 external_mini_cluster.cc:1428] Setting key b626177b86018674fda5ac741e61c76e
I20260926 03:38:03.375895 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: Bootstrap starting.
I20260926 03:38:03.379629 6092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:03.380923 6092 log.cc:824] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:38:03 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:03.384037 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: No bootstrap required, opened a new log
I20260926 03:38:03.389920 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:03.369418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.1.221.190:40783)
0926 03:38:03.370201 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:03.370213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:03.370257 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:38:03.371883 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:03.371906 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:03.372312 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:03.372641 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:03.374415 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.375735 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.377474 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.378169 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.378297 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:03.382874 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:03.382882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:03.382897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:03.382930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:03.385801 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:03.386572 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:03.386578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:03.386585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:03.386798 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:03.387181 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:03.387185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:03.387187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:03.387769 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:03.387990 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:03.388434 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 03:38:03.388835 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260926 03:38:03.393030 6092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } }
I20260926 03:38:03.393383 6092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:03.393468 6092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0f8574a3844d3c84bbb09d71aafb7f, State: Initialized, Role: FOLLOWER
I20260926 03:38:03.394035 6092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } }
I20260926 03:38:03.394255 6092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:03.394375 6092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:03.394519 6092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:03.396651 6092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } }
I20260926 03:38:03.397151 6092 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0f8574a3844d3c84bbb09d71aafb7f; no voters:
I20260926 03:38:03.397686 6092 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:03.397799 6097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:03.399012 6097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [term 1 LEADER]: Becoming Leader. State: Replica: 9a0f8574a3844d3c84bbb09d71aafb7f, State: Running, Role: LEADER
I20260926 03:38:03.399663 6097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } }
I20260926 03:38:03.400022 6092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:03.402478 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } } }
I20260926 03:38:03.402832 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:03.403546 6099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0f8574a3844d3c84bbb09d71aafb7f. Latest consensus state: current_term: 1 leader_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0f8574a3844d3c84bbb09d71aafb7f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40783 } } }
I20260926 03:38:03.403903 6099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:03.404520 6105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:03.409796 6105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:03.420681 6105 catalog_manager.cc:1347] Generated new cluster ID: db779ded983c436383adacb21ff3b35e
I20260926 03:38:03.420792 6105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:03.429281 6105 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:03.430608 6105 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:03.438941 6105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: Generated new TSK 0
I20260926 03:38:03.440075 6105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:38:03.494982 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:03.806082 6118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:03.806367 6118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:03.806489 6118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:03.813915 6118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:03.814042 6118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:03.814195 6118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:03.824116 6118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:03.825444 6118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:03.826700 6118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:03.839125 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:03.839107 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:03.839685 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:03.839924 6118 server_base.cc:1034] running on GCE node
I20260926 03:38:03.840504 6118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:03.841401 6118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:03.842607 6118 hybrid_clock.cc:648] HybridClock initialized: now 1790393883842552 us; error 43 us; skew 500 ppm
I20260926 03:38:03.842928 6118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:03 dist-test-slave-lqrk krb5kdc[6013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393883, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:03.845947 6118 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:38:03.847078 6118 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:03.847180 6118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:03.847513 6118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:03.850417 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "950f439dd35d4af894559165ada7a793"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "3b4896a4b411243a5eb5d3877f50372f"
server_key_iv: "f1d4483f30d37de1646da707e1ad0451"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.851279 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "950f439dd35d4af894559165ada7a793"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "3b4896a4b411243a5eb5d3877f50372f"
server_key_iv: "f1d4483f30d37de1646da707e1ad0451"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.856509 6118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:38:03.860248 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:03.861650 6118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:03.861819 6118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "950f439dd35d4af894559165ada7a793"
format_stamp: "Formatted at 2026-09-26 03:38:03 on dist-test-slave-lqrk"
server_key: "3b4896a4b411243a5eb5d3877f50372f"
server_key_iv: "f1d4483f30d37de1646da707e1ad0451"
server_key_version: "encryptionkey@0"
I20260926 03:38:03.862027 6118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:03.896775 6118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:03.901047 6118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:03.901290 6118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:03.902235 6118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:03.903544 6118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:03.903646 6118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:03.903746 6118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:03.903816 6118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:03.932456 6118 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:37659
I20260926 03:38:03.932518 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:37659 every 8 connection(s)
I20260926 03:38:03.933267 6118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:03.934522 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6118
I20260926 03:38:03.934737 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:03.935456 1910 external_mini_cluster.cc:1428] Setting key 1162bc8e9e3b0e10749ff9ad557a1d05
Sep 26 03:38:03 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393883, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:03.954181 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:03.935819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39741 (local address 127.1.221.190:40783)
0926 03:38:03.936252 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:03.936259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:03.937362 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 03:38:03.939857 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:03.939861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:03.939928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:03.940018 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:03.941876 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.942579 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.944435 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.944987 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.945138 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:03.948250 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:03.948257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:03.948261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:03.948298 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:03.951153 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:03.951871 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:03.951878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:03.951881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:03.952030 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:03.952429 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:03.952434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:03.952437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:03.952792 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:03.952929 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:03.953586 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 03:38:03.953803 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:38:03.954886 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:03.936424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40783 (local address 127.1.221.129:39741)
0926 03:38:03.937189 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:03.937237 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:03.939560 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:03.940263 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:03.940274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:03.940932 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:03.941704 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:03.941738 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.942711 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.942716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:03.944298 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:03.944310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:03.945163 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:03.945168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:03.945348 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:03.945954 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:03.945975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:03.948038 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:03.951280 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:03.951286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:03.951300 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:03.951749 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:03.952149 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:03.952152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:03.952155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:03.952325 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:03.952901 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:03.952906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:03.953206 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 03:38:03.953424 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 03:38:03.953804 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260926 03:38:03.956321 6245 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40783
I20260926 03:38:03.956655 6245 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:38:03.957345 6245 heartbeater.cc:511] Master 127.1.221.190:40783 requested a full tablet report, sending...
I20260926 03:38:03.959429 6057 ts_manager.cc:194] Registered new tserver with Master: 950f439dd35d4af894559165ada7a793 (127.1.221.129:37659)
I20260926 03:38:03.960813 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:39741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:38:03.979245 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:04.289707 6251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:04.289990 6251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:04.290126 6251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:04.297339 6251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:04.297451 6251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:04.297607 6251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:04.308001 6251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:04.309322 6251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:04.310635 6251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:04.322940 6257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:04.322881 6259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:04.322899 6256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:04.323602 6251 server_base.cc:1034] running on GCE node
I20260926 03:38:04.324215 6251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:04.325094 6251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:04.326287 6251 hybrid_clock.cc:648] HybridClock initialized: now 1790393884326231 us; error 43 us; skew 500 ppm
I20260926 03:38:04.326589 6251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:04 dist-test-slave-lqrk krb5kdc[6013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393884, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:04.329576 6251 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:38:04.330639 6251 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:04.330745 6251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:04.331027 6251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:04.333892 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "c783a71190054bff814701eae4011aba"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "1a72e73ce9836fe7533874c7d1645a21"
server_key_iv: "cd046744ba62387f4043c9d922a2254a"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.334667 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "c783a71190054bff814701eae4011aba"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "1a72e73ce9836fe7533874c7d1645a21"
server_key_iv: "cd046744ba62387f4043c9d922a2254a"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.339555 6251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:04.342794 6265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.344038 6251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:04.344177 6251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "c783a71190054bff814701eae4011aba"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "1a72e73ce9836fe7533874c7d1645a21"
server_key_iv: "cd046744ba62387f4043c9d922a2254a"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.344362 6251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:04.365640 6251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:04.369609 6251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:04.369861 6251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:04.371094 6251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:04.372411 6251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:04.372503 6251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.372609 6251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:04.372685 6251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.401070 6251 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:46527
I20260926 03:38:04.401124 6377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:46527 every 8 connection(s)
I20260926 03:38:04.401921 6251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:04.405367 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6251
I20260926 03:38:04.405560 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:04.406080 1910 external_mini_cluster.cc:1428] Setting key 3058cd16c3a945cd79125eedfb4e700b
Sep 26 03:38:04 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393884, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:04.421382 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.404259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52635 (local address 127.1.221.190:40783)
0926 03:38:04.404419 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.404424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.405567 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 03:38:04.407463 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.407468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.407554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.407639 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:04.409186 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.409979 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.411541 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.411940 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.412050 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.415886 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:04.415891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:04.415895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:04.415928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:04.418264 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.419070 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.419073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.419076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.419189 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.419668 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.419671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.419673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.419960 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:04.420068 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.420947 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.421120 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:38:04.422885 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.404710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40783 (local address 127.1.221.130:52635)
0926 03:38:04.405406 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:04.405451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:04.407193 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:04.407882 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:04.407891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:04.408408 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:04.409008 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.409025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.410128 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.410132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.411399 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.411415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.412151 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.412159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.412374 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.413008 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:04.413038 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:04.415663 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:04.418396 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:04.418404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:04.418421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:04.418968 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:04.419298 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:04.419302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:04.419306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:04.419558 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:04.420080 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:04.420088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:04.420550 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 03:38:04.420869 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 03:38:04.421432 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:38:04.424641 6378 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40783
I20260926 03:38:04.424998 6378 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:04.425899 6378 heartbeater.cc:511] Master 127.1.221.190:40783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:38:04.427742 6057 ts_manager.cc:194] Registered new tserver with Master: c783a71190054bff814701eae4011aba (127.1.221.130:46527)
I20260926 03:38:04.428782 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:52635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:38:04.450631 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:04.761473 6383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:04.761758 6383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:04.761879 6383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:04.769173 6383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:04.769286 6383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:04.769439 6383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:04.779248 6383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:04.780593 6383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:04.781890 6383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:04.793943 6389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:04.793970 6391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:04.794023 6388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:04.794914 6383 server_base.cc:1034] running on GCE node
I20260926 03:38:04.795516 6383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:04.796401 6383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:04.797581 6383 hybrid_clock.cc:648] HybridClock initialized: now 1790393884797540 us; error 34 us; skew 500 ppm
I20260926 03:38:04.797875 6383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:04 dist-test-slave-lqrk krb5kdc[6013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393884, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:04.804591 6383 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:38:04.805696 6383 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:04.805804 6383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:04.806099 6383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:04.809142 6383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "de82480671134d78a2e59c67eb24d0d6"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "2449773568d2a01788081a3a254d5a2e"
server_key_iv: "7a6245f06c7ffc9efd0197049fc61155"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.809947 6383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "de82480671134d78a2e59c67eb24d0d6"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "2449773568d2a01788081a3a254d5a2e"
server_key_iv: "7a6245f06c7ffc9efd0197049fc61155"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.814855 6383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:38:04.818193 6397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.819463 6383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:04.819615 6383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "de82480671134d78a2e59c67eb24d0d6"
format_stamp: "Formatted at 2026-09-26 03:38:04 on dist-test-slave-lqrk"
server_key: "2449773568d2a01788081a3a254d5a2e"
server_key_iv: "7a6245f06c7ffc9efd0197049fc61155"
server_key_version: "encryptionkey@0"
I20260926 03:38:04.819844 6383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:04.834192 6383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:04.838119 6383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:04.838348 6383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:04.839195 6383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:04.840579 6383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:04.840670 6383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.840772 6383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:04.840840 6383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:04.868175 6383 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:33195
I20260926 03:38:04.868239 6509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:33195 every 8 connection(s)
I20260926 03:38:04.868980 6383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:04.878296 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6383
I20260926 03:38:04.878474 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:04.878945 1910 external_mini_cluster.cc:1428] Setting key 0e635d1f42f88a3da22230100f677004
Sep 26 03:38:04 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393884, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:04.887296 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.871225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55497 (local address 127.1.221.190:40783)
0926 03:38:04.871435 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.871441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.872686 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 03:38:04.874364 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.874368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.874437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.874521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:04.875954 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.876660 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.878224 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.878671 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.878763 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.882653 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:04.882665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:04.882669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:04.882698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:04.884917 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.885544 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.885548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.885550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.885654 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.885977 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.885980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.885982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.886220 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:04.886339 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.886900 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.887064 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:38:04.888288 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.871833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40783 (local address 127.1.221.131:55497)
0926 03:38:04.872520 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:04.872556 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:04.874108 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:04.874703 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:04.874710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:04.875210 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:04.875806 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.875824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.876777 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.876780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.878066 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.878080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.878881 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.878889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.879144 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.879919 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:04.879953 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:04.882470 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:04.885034 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:04.885040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:04.885053 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:04.885454 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:04.885740 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:04.885743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:04.885745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:04.885901 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:04.886312 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:04.886317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:04.886612 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 03:38:04.886823 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 03:38:04.887194 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":280,"threads_started":1}
I20260926 03:38:04.889606 6510 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40783
I20260926 03:38:04.889853 6510 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:04.890450 6510 heartbeater.cc:511] Master 127.1.221.190:40783 requested a full tablet report, sending...
I20260926 03:38:04.891709 6057 ts_manager.cc:194] Registered new tserver with Master: de82480671134d78a2e59c67eb24d0d6 (127.1.221.131:33195)
I20260926 03:38:04.892583 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:55497
I20260926 03:38:04.894244 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:38:04.910233 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.898389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.1.221.190:40783)
0926 03:38:04.898594 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.898601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.898990 (+ 389us) server_negotiation.cc:407] Connection header received
0926 03:38:04.899115 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.899118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.899187 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.899262 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:04.900682 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.901442 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.902605 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.903033 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.903125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.905573 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:04.905578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:04.905581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:04.905608 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:04.907949 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.908584 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.908589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.908593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.908725 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:04.909015 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:04.909017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:04.909019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:04.909345 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:04.909540 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.909766 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.909916 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:38:04.914074 6056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:04.920665 6056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:04.942830 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.933367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.1.221.130:46527)
0926 03:38:04.934889 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.934895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.934921 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:38:04.935015 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.935024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.935247 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.936207 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:04.936398 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.937534 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.940289 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.941716 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.941839 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.942046 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.942325 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.942494 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1392,"thread_start_us":1275,"threads_started":1}
I20260926 03:38:04.942822 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.932777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:46527 (local address 127.0.0.1:38982)
0926 03:38:04.933674 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:04.933719 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:04.933872 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:04.935503 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:04.935510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:04.935550 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:04.935988 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.936001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.937656 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.937661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.940114 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.940127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.941906 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.941912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.942117 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.942158 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:38:04.942263 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:38:04.942453 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":777,"threads_started":1}
I20260926 03:38:04.946621 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.932803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:33195 (local address 127.0.0.1:47492)
0926 03:38:04.938140 (+ 5337us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:04.938159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:04.938363 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:04.939268 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:04.939273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:04.939296 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:04.939792 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.939804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.941742 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.941747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.944085 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.944099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.945899 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.945903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.946014 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.946035 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 03:38:04.946102 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:38:04.946240 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5196,"mutex_wait_us":1,"thread_start_us":5046,"threads_started":1}
I20260926 03:38:04.946653 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.933353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.1.221.131:33195)
0926 03:38:04.934486 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.934496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.938256 (+ 3760us) server_negotiation.cc:407] Connection header received
0926 03:38:04.938564 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.938573 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.938816 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.938973 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:04.940010 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.941603 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.944276 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.945526 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.945608 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.945764 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.946192 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.946359 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":787,"threads_started":1}
I20260926 03:38:04.946969 6313 tablet_service.cc:1467] Processing CreateTablet for tablet ca9866a2e57243cb9d49153109044022 (DEFAULT_TABLE table=test-table [id=2dc50817c3094556b5be4fb65a87bba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:04.947795 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.935786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37659 (local address 127.0.0.1:40736)
0926 03:38:04.939881 (+ 4095us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:04.939898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:04.940015 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:04.940821 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:04.940826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:04.940849 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:04.941129 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.941136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.942783 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.942789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.945009 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:04.945024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.946996 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.947002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:04.947173 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.947204 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:38:04.947295 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:38:04.947442 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3956,"mutex_wait_us":3,"thread_start_us":3780,"threads_started":1}
I20260926 03:38:04.947853 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:04.936294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.1.221.129:37659)
0926 03:38:04.936903 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:04.936912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:04.940141 (+ 3229us) server_negotiation.cc:407] Connection header received
0926 03:38:04.940243 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:04.940251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:04.940469 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:04.940647 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:04.941283 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.942602 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.945181 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:04.946525 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:04.946652 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:04.946860 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:04.947366 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:38:04.947547 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
I20260926 03:38:04.948572 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9866a2e57243cb9d49153109044022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:04.950423 6445 tablet_service.cc:1467] Processing CreateTablet for tablet ca9866a2e57243cb9d49153109044022 (DEFAULT_TABLE table=test-table [id=2dc50817c3094556b5be4fb65a87bba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:04.951586 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9866a2e57243cb9d49153109044022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:04.958671 6180 tablet_service.cc:1467] Processing CreateTablet for tablet ca9866a2e57243cb9d49153109044022 (DEFAULT_TABLE table=test-table [id=2dc50817c3094556b5be4fb65a87bba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:04.960194 6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9866a2e57243cb9d49153109044022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:04.960772 6528 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Bootstrap starting.
I20260926 03:38:04.963546 6245 heartbeater.cc:503] Master 127.1.221.190:40783 was elected leader, sending a full tablet report...
I20260926 03:38:04.964674 6528 tablet_bootstrap.cc:654] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:04.966056 6528 log.cc:824] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:04.968988 6528 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: No bootstrap required, opened a new log
I20260926 03:38:04.969237 6528 ts_tablet_manager.cc:1397] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:38:04.972306 6531 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Bootstrap starting.
I20260926 03:38:04.972654 6529 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Bootstrap starting.
I20260926 03:38:04.975317 6531 tablet_bootstrap.cc:654] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:04.976032 6529 tablet_bootstrap.cc:654] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:04.976385 6531 log.cc:824] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:04.977259 6529 log.cc:824] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:04.978368 6528 raft_consensus.cc:348] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.978799 6528 raft_consensus.cc:374] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:04.978899 6528 raft_consensus.cc:729] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c783a71190054bff814701eae4011aba, State: Initialized, Role: FOLLOWER
I20260926 03:38:04.979234 6531 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: No bootstrap required, opened a new log
I20260926 03:38:04.979514 6531 ts_tablet_manager.cc:1397] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260926 03:38:04.979566 6528 consensus_queue.cc:260] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.980350 6529 tablet_bootstrap.cc:492] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: No bootstrap required, opened a new log
I20260926 03:38:04.980610 6529 ts_tablet_manager.cc:1397] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260926 03:38:04.980739 6378 heartbeater.cc:503] Master 127.1.221.190:40783 was elected leader, sending a full tablet report...
I20260926 03:38:04.981361 6528 ts_tablet_manager.cc:1428] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 03:38:04.988567 6531 raft_consensus.cc:348] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.988919 6531 raft_consensus.cc:374] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:04.988732 6529 raft_consensus.cc:348] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.989012 6531 raft_consensus.cc:729] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950f439dd35d4af894559165ada7a793, State: Initialized, Role: FOLLOWER
I20260926 03:38:04.989133 6529 raft_consensus.cc:374] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:04.989234 6529 raft_consensus.cc:729] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de82480671134d78a2e59c67eb24d0d6, State: Initialized, Role: FOLLOWER
I20260926 03:38:04.989614 6531 consensus_queue.cc:260] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.989758 6529 consensus_queue.cc:260] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:04.990806 6510 heartbeater.cc:503] Master 127.1.221.190:40783 was elected leader, sending a full tablet report...
I20260926 03:38:04.991336 6529 ts_tablet_manager.cc:1428] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260926 03:38:04.992314 6531 ts_tablet_manager.cc:1428] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260926 03:38:05.023993 6535 raft_consensus.cc:482] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:05.024298 6535 raft_consensus.cc:504] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:05.026010 6535 leader_election.cc:288] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c783a71190054bff814701eae4011aba (127.1.221.130:46527), de82480671134d78a2e59c67eb24d0d6 (127.1.221.131:33195)
I20260926 03:38:05.032452 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.026471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:50203 (local address 127.1.221.130:46527)
0926 03:38:05.026839 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.026844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.027019 (+ 175us) server_negotiation.cc:407] Connection header received
0926 03:38:05.027219 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.027223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.027286 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.027358 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:05.027942 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.028652 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.030543 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.031574 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.031743 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.031817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.032056 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.032190 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260926 03:38:05.032610 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.026274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:46527 (local address 127.1.221.129:50203)
0926 03:38:05.026918 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:05.026934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:05.027083 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:05.027431 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:05.027436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:05.027464 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:05.027800 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:05.027809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.028775 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.028780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:05.030400 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:05.030413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.031759 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.031765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:05.031915 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.031938 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:38:05.032085 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 03:38:05.032227 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":74,"spinlock_wait_cycles":22912,"thread_start_us":298,"threads_started":1}
I20260926 03:38:05.032697 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.026493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36391 (local address 127.1.221.131:33195)
0926 03:38:05.026762 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.026768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.027504 (+ 736us) server_negotiation.cc:407] Connection header received
0926 03:38:05.027673 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.027678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.027738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.027810 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:05.028287 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.028976 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.030794 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.031847 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.031980 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.032047 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.032338 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.032456 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 03:38:05.032881 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.026274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:33195 (local address 127.1.221.129:36391)
0926 03:38:05.027409 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:05.027426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:05.027556 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:05.027836 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:05.027839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:05.027862 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:05.028157 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:05.028166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.029111 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.029115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:05.030654 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:05.030668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.032069 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.032076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:05.032206 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.032228 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:38:05.032303 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:38:05.032428 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":535,"thread_start_us":278,"threads_started":1}
I20260926 03:38:05.033685 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9866a2e57243cb9d49153109044022" candidate_uuid: "950f439dd35d4af894559165ada7a793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c783a71190054bff814701eae4011aba" is_pre_election: true
I20260926 03:38:05.033684 6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9866a2e57243cb9d49153109044022" candidate_uuid: "950f439dd35d4af894559165ada7a793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de82480671134d78a2e59c67eb24d0d6" is_pre_election: true
I20260926 03:38:05.034133 6465 raft_consensus.cc:2454] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950f439dd35d4af894559165ada7a793 in term 0.
I20260926 03:38:05.034132 6333 raft_consensus.cc:2454] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950f439dd35d4af894559165ada7a793 in term 0.
I20260926 03:38:05.034749 6133 leader_election.cc:302] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 950f439dd35d4af894559165ada7a793, c783a71190054bff814701eae4011aba, de82480671134d78a2e59c67eb24d0d6; no voters:
I20260926 03:38:05.035135 6535 raft_consensus.cc:2781] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:05.035297 6535 raft_consensus.cc:482] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:05.035427 6535 raft_consensus.cc:3037] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:05.037537 6535 raft_consensus.cc:504] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:05.038121 6535 leader_election.cc:288] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [CANDIDATE]: Term 1 election: Requested vote from peers c783a71190054bff814701eae4011aba (127.1.221.130:46527), de82480671134d78a2e59c67eb24d0d6 (127.1.221.131:33195)
I20260926 03:38:05.038550 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9866a2e57243cb9d49153109044022" candidate_uuid: "950f439dd35d4af894559165ada7a793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c783a71190054bff814701eae4011aba"
I20260926 03:38:05.038584 6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9866a2e57243cb9d49153109044022" candidate_uuid: "950f439dd35d4af894559165ada7a793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de82480671134d78a2e59c67eb24d0d6"
I20260926 03:38:05.038741 6333 raft_consensus.cc:3037] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:05.038756 6465 raft_consensus.cc:3037] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:05.040887 6333 raft_consensus.cc:2454] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950f439dd35d4af894559165ada7a793 in term 1.
I20260926 03:38:05.040899 6465 raft_consensus.cc:2454] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950f439dd35d4af894559165ada7a793 in term 1.
I20260926 03:38:05.041261 6133 leader_election.cc:302] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 950f439dd35d4af894559165ada7a793, c783a71190054bff814701eae4011aba; no voters:
I20260926 03:38:05.041522 6535 raft_consensus.cc:2781] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:05.041910 6535 raft_consensus.cc:686] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 1 LEADER]: Becoming Leader. State: Replica: 950f439dd35d4af894559165ada7a793, State: Running, Role: LEADER
I20260926 03:38:05.042364 6535 consensus_queue.cc:237] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } }
I20260926 03:38:05.046873 6056 catalog_manager.cc:5495] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 reported cstate change: term changed from 0 to 1, leader changed from <none> to 950f439dd35d4af894559165ada7a793 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "950f439dd35d4af894559165ada7a793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "950f439dd35d4af894559165ada7a793" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37659 } health_report { overall_health: HEALTHY } } }
I20260926 03:38:05.066888 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.061922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.1.221.129:37659)
0926 03:38:05.062134 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.062139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.062158 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:05.062334 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.062337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.062414 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.062535 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:05.062840 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.063567 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.064781 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.065261 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.065351 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.065771 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:05.066354 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:05.066461 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.066557 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.066625 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:38:05.074703 6333 raft_consensus.cc:1261] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 1 FOLLOWER]: Refusing update from remote peer 950f439dd35d4af894559165ada7a793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:05.074710 6465 raft_consensus.cc:1261] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 1 FOLLOWER]: Refusing update from remote peer 950f439dd35d4af894559165ada7a793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:05.075659 6539 consensus_queue.cc:1035] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de82480671134d78a2e59c67eb24d0d6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:05.075989 6535 consensus_queue.cc:1035] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c783a71190054bff814701eae4011aba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:05.086776 6542 mvcc.cc:205] Tried to move back new op lower bound from 7333453353256730624 to 7333453353137045504. Current Snapshot: MvccSnapshot[applied={T|T < 7333453353256730624}]
I20260926 03:38:05.103639 6056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:05.103876 6056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:05.107460 6056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a0f8574a3844d3c84bbb09d71aafb7f: Sending DeleteTablet for 3 replicas of tablet ca9866a2e57243cb9d49153109044022
I20260926 03:38:05.108695 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet ca9866a2e57243cb9d49153109044022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:05 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:47492
I20260926 03:38:05.108705 6313 tablet_service.cc:1514] Processing DeleteTablet for tablet ca9866a2e57243cb9d49153109044022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:05 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38982
I20260926 03:38:05.109460 6180 tablet_service.cc:1514] Processing DeleteTablet for tablet ca9866a2e57243cb9d49153109044022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:05 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40736
I20260926 03:38:05.109674 6552 tablet_replica.cc:330] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: stopping tablet replica
I20260926 03:38:05.110167 6552 raft_consensus.cc:2229] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:05.110585 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6118
I20260926 03:38:05.111004 6552 raft_consensus.cc:2258] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:05.111589 6553 tablet_replica.cc:330] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: stopping tablet replica
I20260926 03:38:05.112357 6553 raft_consensus.cc:2229] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:05.112987 6553 raft_consensus.cc:2258] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:05.114497 6552 ts_tablet_manager.cc:1905] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:05.120998 6554 tablet_replica.cc:330] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: stopping tablet replica
I20260926 03:38:05.121510 6554 raft_consensus.cc:2229] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:05.122222 6554 raft_consensus.cc:2258] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:05.123353 6553 ts_tablet_manager.cc:1905] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:05.124583 6554 ts_tablet_manager.cc:1905] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:05.131140 6552 ts_tablet_manager.cc:1918] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:05.131508 6552 log.cc:1196] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/ca9866a2e57243cb9d49153109044022
I20260926 03:38:05.132153 6552 ts_tablet_manager.cc:1939] T ca9866a2e57243cb9d49153109044022 P de82480671134d78a2e59c67eb24d0d6: Deleting consensus metadata
I20260926 03:38:05.132562 6554 ts_tablet_manager.cc:1918] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:05.132740 6554 log.cc:1196] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/ca9866a2e57243cb9d49153109044022
I20260926 03:38:05.133252 6554 ts_tablet_manager.cc:1939] T ca9866a2e57243cb9d49153109044022 P 950f439dd35d4af894559165ada7a793: Deleting consensus metadata
Sep 26 03:38:05 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:05.133612 6553 ts_tablet_manager.cc:1918] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:05.133854 6553 log.cc:1196] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/ca9866a2e57243cb9d49153109044022
I20260926 03:38:05.135054 6041 catalog_manager.cc:4841] TS 950f439dd35d4af894559165ada7a793 (127.1.221.129:37659): tablet ca9866a2e57243cb9d49153109044022 (table test-table [id=2dc50817c3094556b5be4fb65a87bba1]) successfully deleted
I20260926 03:38:05.135178 6042 catalog_manager.cc:4841] TS de82480671134d78a2e59c67eb24d0d6 (127.1.221.131:33195): tablet ca9866a2e57243cb9d49153109044022 (table test-table [id=2dc50817c3094556b5be4fb65a87bba1]) successfully deleted
I20260926 03:38:05.135874 6553 ts_tablet_manager.cc:1939] T ca9866a2e57243cb9d49153109044022 P c783a71190054bff814701eae4011aba: Deleting consensus metadata
I20260926 03:38:05.137481 6041 catalog_manager.cc:4841] TS c783a71190054bff814701eae4011aba (127.1.221.130:46527): tablet ca9866a2e57243cb9d49153109044022 (table test-table [id=2dc50817c3094556b5be4fb65a87bba1]) successfully deleted
I20260926 03:38:05.140465 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.113655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.1.221.129:37659)
0926 03:38:05.113841 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.113848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.123354 (+ 9506us) server_negotiation.cc:407] Connection header received
0926 03:38:05.123765 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.123770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.123860 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.123932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:05.125689 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.126861 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.128685 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.129362 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.129460 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.134518 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:05.134530 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:05.134548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:05.134596 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:05.137819 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.138451 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.138458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.138461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.138611 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.138957 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.138961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.138963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.139305 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:05.139600 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.139928 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.140165 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:38:05.141021 6240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:05.239363 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6118
I20260926 03:38:05.253978 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6251
Sep 26 03:38:05 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:38:05.267908 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.254762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.1.221.130:46527)
0926 03:38:05.254943 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.254948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.254968 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:38:05.255089 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.255093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.255173 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.255259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:05.256463 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.257225 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.258416 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.258822 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.258929 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.262736 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:05.262753 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:05.262770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:05.262815 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:05.265362 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.265977 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.265983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.265986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.266127 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.266545 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.266548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.266550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.266925 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:05.267133 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.267463 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.267642 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:38:05.268604 6373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:05.354532 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6251
I20260926 03:38:05.371629 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6383
Sep 26 03:38:05 dist-test-slave-lqrk krb5kdc[6013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:38:05.387925 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:05.372633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.1.221.131:33195)
0926 03:38:05.372780 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:05.372784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:05.372801 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:38:05.372853 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:05.372857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:05.372926 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:05.373006 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:05.374521 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.375266 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.377050 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:05.377429 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:05.377516 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:05.382763 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:05.382780 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:05.382797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:05.382840 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:05.385430 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.386100 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.386108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.386110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.386254 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:05.386544 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:05.386546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:05.386548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:05.386901 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:05.387109 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:05.387354 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:38:05.387604 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:38:05.388585 6505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:05.476577 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6383
I20260926 03:38:05.489259 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6026
I20260926 03:38:05.490375 6087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:05.595875 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6026
2026-09-26T03:38:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3821 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T03:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:05Z Disabled control of system clock
I20260926 03:38:05.656620 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41475
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:38:05.968638 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:05.968935 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:05.969110 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:05.976401 6563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:05.976507 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:05.976570 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:05.976646 6563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:05.976707 6563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:05.986086 6563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:05.987412 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:05.989194 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:06.000545 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.000576 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.000651 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:06.001387 6563 server_base.cc:1034] running on GCE node
I20260926 03:38:06.002221 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:06.003750 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:06.004984 6563 hybrid_clock.cc:648] HybridClock initialized: now 1790393886004939 us; error 29 us; skew 500 ppm
I20260926 03:38:06.005334 6563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:06.007688 6563 webserver.cc:460] Webserver started at http://127.1.221.190:33117/ using document root <none> and password file <none>
I20260926 03:38:06.008404 6563 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:06.008507 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:06.008821 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:06.011848 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "f5bc9594f2f741e8b4f569a79ca0b39e"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "117302ebbfdfa5189ed0730be704588b"
server_key_iv: "169c322a6b9485d7c0dbe22f1eb1f3f6"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.012687 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "f5bc9594f2f741e8b4f569a79ca0b39e"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "117302ebbfdfa5189ed0730be704588b"
server_key_iv: "169c322a6b9485d7c0dbe22f1eb1f3f6"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.017880 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:38:06.021375 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.022745 6563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:06.022903 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "f5bc9594f2f741e8b4f569a79ca0b39e"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "117302ebbfdfa5189ed0730be704588b"
server_key_iv: "169c322a6b9485d7c0dbe22f1eb1f3f6"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.023113 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:06.028632 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:06.029467 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:06.029750 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:06.046555 6563 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41475
I20260926 03:38:06.046561 6628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41475 every 8 connection(s)
I20260926 03:38:06.047966 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:06.051716 6629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:06.054473 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6563
I20260926 03:38:06.054651 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:06.055444 1910 external_mini_cluster.cc:1428] Setting key 3b5928c195f58f32b4fa5921cd2e72a1
I20260926 03:38:06.063725 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:06.057569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.1.221.190:41475)
0926 03:38:06.058565 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:06.058576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:06.058624 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:38:06.060243 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:06.060266 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:06.060716 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:06.061116 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:06.062095 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:06.062110 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:06.062122 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:06.062137 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:06.062197 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:06.062214 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:06.062412 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:06.062538 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 03:38:06.062789 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":341,"threads_started":1}
I20260926 03:38:06.063753 6629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: Bootstrap starting.
I20260926 03:38:06.066802 6629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:06.067818 6629 log.cc:824] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:06.070271 6629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: No bootstrap required, opened a new log
I20260926 03:38:06.077433 6629 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } }
I20260926 03:38:06.077777 6629 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:06.077870 6629 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc9594f2f741e8b4f569a79ca0b39e, State: Initialized, Role: FOLLOWER
I20260926 03:38:06.078496 6629 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } }
I20260926 03:38:06.078748 6629 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:06.078887 6629 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:06.079046 6629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:06.081125 6629 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } }
I20260926 03:38:06.081558 6629 leader_election.cc:302] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5bc9594f2f741e8b4f569a79ca0b39e; no voters:
I20260926 03:38:06.082083 6629 leader_election.cc:288] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:06.082203 6634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:06.083636 6634 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [term 1 LEADER]: Becoming Leader. State: Replica: f5bc9594f2f741e8b4f569a79ca0b39e, State: Running, Role: LEADER
I20260926 03:38:06.084178 6629 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:06.084117 6634 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } }
I20260926 03:38:06.086598 6636 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5bc9594f2f741e8b4f569a79ca0b39e. Latest consensus state: current_term: 1 leader_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } } }
I20260926 03:38:06.086984 6636 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:06.086882 6635 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bc9594f2f741e8b4f569a79ca0b39e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41475 } } }
I20260926 03:38:06.087169 6635 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:06.087874 6643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:06.092612 6643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:06.100489 6643 catalog_manager.cc:1347] Generated new cluster ID: 7aae5eb652d14c8daed830551de354b8
I20260926 03:38:06.100617 6643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:06.130559 6643 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:06.131002 6643 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:06.143756 6643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: Generated new TSK 0
I20260926 03:38:06.144564 6643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:06.150151 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--builtin_ntp_servers=127.1.221.148:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:38:06.464661 6653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:06.464922 6653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:06.465045 6653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:06.472277 6653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:06.472371 6653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:06.472512 6653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:06.481576 6653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:06.482801 6653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:06.484525 6653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:06.496737 6658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.496706 6661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.496765 6659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:06.497757 6653 server_base.cc:1034] running on GCE node
I20260926 03:38:06.498313 6653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:06.499212 6653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:06.500398 6653 hybrid_clock.cc:648] HybridClock initialized: now 1790393886500346 us; error 43 us; skew 500 ppm
I20260926 03:38:06.500699 6653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:06.502770 6653 webserver.cc:460] Webserver started at http://127.1.221.129:44551/ using document root <none> and password file <none>
I20260926 03:38:06.503453 6653 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:06.503571 6653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:06.503868 6653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:06.506589 6653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "2da7504e33784cf5be16a31cf027642f"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "d057f6903511d11bf189fd9e99ac1d82"
server_key_iv: "449bfafb3b7e3c368d0377811487a270"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.507318 6653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "2da7504e33784cf5be16a31cf027642f"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "d057f6903511d11bf189fd9e99ac1d82"
server_key_iv: "449bfafb3b7e3c368d0377811487a270"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.512238 6653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:38:06.515616 6667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.516875 6653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:38:06.517019 6653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "2da7504e33784cf5be16a31cf027642f"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "d057f6903511d11bf189fd9e99ac1d82"
server_key_iv: "449bfafb3b7e3c368d0377811487a270"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.517223 6653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:06.522454 6653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:06.523202 6653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:06.523484 6653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:06.524328 6653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:06.525630 6653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:06.525731 6653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.525835 6653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:06.525904 6653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.554276 6653 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:45413
I20260926 03:38:06.554325 6779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:45413 every 8 connection(s)
I20260926 03:38:06.555569 6653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:06.559558 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6653
I20260926 03:38:06.559769 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:06.560552 1910 external_mini_cluster.cc:1428] Setting key fa7ddcba1f3bfb31dba3d7b4b38637a8
I20260926 03:38:06.563402 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--builtin_ntp_servers=127.1.221.148:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:38:06.564110 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:06.557941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38477 (local address 127.1.221.190:41475)
0926 03:38:06.558103 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:06.558108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:06.559593 (+ 1485us) server_negotiation.cc:407] Connection header received
0926 03:38:06.561775 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:06.561780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:06.561843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:06.561930 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:06.563442 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:06.563446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:06.563449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:06.563465 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:06.563499 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:06.563513 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:06.563628 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:06.563863 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:38:06.563909 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:38:06.565516 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:06.558394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41475 (local address 127.1.221.129:38477)
0926 03:38:06.559425 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:06.559473 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:06.561501 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:06.562195 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:06.562206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:06.562323 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:06.563045 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:06.563048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:06.563101 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:06.563226 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:06.563676 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:06.563686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:06.563693 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:38:06.563952 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 03:38:06.564325 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":256,"threads_started":1}
I20260926 03:38:06.566933 6780 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41475
I20260926 03:38:06.567248 6780 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:06.567936 6780 heartbeater.cc:511] Master 127.1.221.190:41475 requested a full tablet report, sending...
I20260926 03:38:06.570222 6594 ts_manager.cc:194] Registered new tserver with Master: 2da7504e33784cf5be16a31cf027642f (127.1.221.129:45413)
W20260926 03:38:06.871784 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:06.872078 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:06.872164 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:06.879248 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:06.879350 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:06.879521 6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:06.888571 6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:06.889778 6783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:06.891539 6783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:06.903996 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.904032 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:06.903962 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:06.905007 6783 server_base.cc:1034] running on GCE node
I20260926 03:38:06.905627 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:06.906538 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:06.907730 6783 hybrid_clock.cc:648] HybridClock initialized: now 1790393886907681 us; error 38 us; skew 500 ppm
I20260926 03:38:06.908037 6783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:06.910166 6783 webserver.cc:460] Webserver started at http://127.1.221.130:42127/ using document root <none> and password file <none>
I20260926 03:38:06.910812 6783 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:06.910910 6783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:06.911207 6783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:06.913985 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "4c7b2ef47c97434c82f0c10855af92b4"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "afbfc81f1c4129ca72fdbc48958054be"
server_key_iv: "d83fed12ea0a25dfb2a88abb93df3e7f"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.914763 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "4c7b2ef47c97434c82f0c10855af92b4"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "afbfc81f1c4129ca72fdbc48958054be"
server_key_iv: "d83fed12ea0a25dfb2a88abb93df3e7f"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.919708 6783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:06.923069 6797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.924450 6783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:06.924599 6783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "4c7b2ef47c97434c82f0c10855af92b4"
format_stamp: "Formatted at 2026-09-26 03:38:06 on dist-test-slave-lqrk"
server_key: "afbfc81f1c4129ca72fdbc48958054be"
server_key_iv: "d83fed12ea0a25dfb2a88abb93df3e7f"
server_key_version: "encryptionkey@0"
I20260926 03:38:06.924806 6783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:06.930346 6783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:06.931113 6783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:06.931347 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:06.932292 6783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:06.933662 6783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:06.933753 6783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.933856 6783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:06.933923 6783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:06.962947 6783 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:35927
I20260926 03:38:06.962983 6909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:35927 every 8 connection(s)
I20260926 03:38:06.964298 6783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:06.971711 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:06.966692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43697 (local address 127.1.221.190:41475)
0926 03:38:06.966861 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:06.966866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:06.968248 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 03:38:06.969832 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:06.969836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:06.969888 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:06.969989 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:06.971122 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:06.971127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:06.971130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:06.971145 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:06.971178 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:06.971190 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:06.971279 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:06.971440 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 03:38:06.971493 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:38:06.972206 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6783
I20260926 03:38:06.972362 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:06.972810 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:06.967177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41475 (local address 127.1.221.130:43697)
0926 03:38:06.968073 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:06.968109 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:06.969568 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:06.970181 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:06.970189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:06.970276 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:06.970851 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:06.970854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:06.970889 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:06.970971 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:06.971262 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:06.971268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:06.971273 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:38:06.971494 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 03:38:06.971686 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":290,"threads_started":1}
I20260926 03:38:06.972935 1910 external_mini_cluster.cc:1428] Setting key 8595e235366b03e058d79662bfaa7e94
I20260926 03:38:06.974154 6910 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41475
I20260926 03:38:06.974450 6910 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:06.975101 6910 heartbeater.cc:511] Master 127.1.221.190:41475 requested a full tablet report, sending...
I20260926 03:38:06.975754 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--builtin_ntp_servers=127.1.221.148:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:38:06.976624 6594 ts_manager.cc:194] Registered new tserver with Master: 4c7b2ef47c97434c82f0c10855af92b4 (127.1.221.130:35927)
W20260926 03:38:07.283311 6913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:07.283617 6913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:07.283744 6913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:07.290975 6913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:07.291074 6913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:07.291217 6913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:07.300217 6913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:07.301441 6913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:07.303164 6913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:07.315451 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:07.315482 6919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:07.315505 6921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:07.316260 6913 server_base.cc:1034] running on GCE node
I20260926 03:38:07.316836 6913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:07.317713 6913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:07.318908 6913 hybrid_clock.cc:648] HybridClock initialized: now 1790393887318854 us; error 48 us; skew 500 ppm
I20260926 03:38:07.319205 6913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:07.321376 6913 webserver.cc:460] Webserver started at http://127.1.221.131:38261/ using document root <none> and password file <none>
I20260926 03:38:07.322036 6913 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:07.322130 6913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:07.322414 6913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:07.325224 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "a7596e55353f483ba6ef632c33506705"
format_stamp: "Formatted at 2026-09-26 03:38:07 on dist-test-slave-lqrk"
server_key: "8184e8b32a3fdad434bc8ed6db74992a"
server_key_iv: "cb4c86a26948d44e6f76b4d136f158ab"
server_key_version: "encryptionkey@0"
I20260926 03:38:07.325999 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "a7596e55353f483ba6ef632c33506705"
format_stamp: "Formatted at 2026-09-26 03:38:07 on dist-test-slave-lqrk"
server_key: "8184e8b32a3fdad434bc8ed6db74992a"
server_key_iv: "cb4c86a26948d44e6f76b4d136f158ab"
server_key_version: "encryptionkey@0"
I20260926 03:38:07.331130 6913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:07.334718 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:07.336155 6913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:07.336314 6913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "a7596e55353f483ba6ef632c33506705"
format_stamp: "Formatted at 2026-09-26 03:38:07 on dist-test-slave-lqrk"
server_key: "8184e8b32a3fdad434bc8ed6db74992a"
server_key_iv: "cb4c86a26948d44e6f76b4d136f158ab"
server_key_version: "encryptionkey@0"
I20260926 03:38:07.336542 6913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:07.342233 6913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:07.342990 6913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:07.343231 6913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:07.344120 6913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:07.345415 6913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:07.345506 6913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:07.345608 6913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:07.345677 6913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:07.374264 6913 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:39199
I20260926 03:38:07.374311 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:39199 every 8 connection(s)
I20260926 03:38:07.375550 6913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:07.382891 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 6913
I20260926 03:38:07.383051 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:07.383159 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.377936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:58049 (local address 127.1.221.190:41475)
0926 03:38:07.378121 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.378125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.379497 (+ 1372us) server_negotiation.cc:407] Connection header received
0926 03:38:07.381120 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.381124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.381177 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.381264 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.382484 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.382490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.382494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.382516 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.382556 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.382572 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.382713 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.382831 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.382898 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:38:07.383575 1910 external_mini_cluster.cc:1428] Setting key abaec2990015f0fe1e96a4fcf15eb300
I20260926 03:38:07.384248 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.378442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41475 (local address 127.1.221.131:58049)
0926 03:38:07.379318 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.379360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.380879 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.381456 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.381464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.381562 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.382191 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.382194 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.382233 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.382323 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.382660 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.382666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.382671 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.382881 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.383128 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":273,"threads_started":1}
I20260926 03:38:07.385555 7040 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41475
I20260926 03:38:07.385834 7040 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:07.386473 7040 heartbeater.cc:511] Master 127.1.221.190:41475 requested a full tablet report, sending...
I20260926 03:38:07.387861 6594 ts_manager.cc:194] Registered new tserver with Master: a7596e55353f483ba6ef632c33506705 (127.1.221.131:39199)
I20260926 03:38:07.398799 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:38:07.405602 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.403336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.1.221.190:41475)
0926 03:38:07.403513 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.403517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.404042 (+ 525us) server_negotiation.cc:407] Connection header received
0926 03:38:07.404190 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.404194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.404243 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.404325 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.405152 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.405156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.405159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.405171 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.405195 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.405205 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.405293 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.405341 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.405386 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:38:07.409803 6594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:07.416445 6594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:07.443357 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.427878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33046)
0926 03:38:07.440093 (+ 12215us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.440143 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.440313 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.440912 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.440919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.441012 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.441959 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.441961 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.441987 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.442059 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.442555 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.442567 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.442576 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.442759 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.442864 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11261,"threads_started":1}
I20260926 03:38:07.444247 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.428053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.1.221.129:45413)
0926 03:38:07.428661 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.428670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.440269 (+ 11599us) server_negotiation.cc:407] Connection header received
0926 03:38:07.440450 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.440459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.440731 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.440892 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.442225 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.442235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.442253 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.442284 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.442351 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.442373 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.442501 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.443781 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.443900 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":1554,"threads_started":1}
I20260926 03:38:07.446213 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.428050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:39199 (local address 127.0.0.1:53336)
0926 03:38:07.442148 (+ 14098us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.442168 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.442544 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.444338 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.444344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.444372 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.445106 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.445109 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.445125 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.445169 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.445664 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.445667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.445670 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.445779 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.445830 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13927,"thread_start_us":10935,"threads_started":1}
I20260926 03:38:07.446231 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.429146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.1.221.131:39199)
0926 03:38:07.436546 (+ 7400us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.436558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.442279 (+ 5721us) server_negotiation.cc:407] Connection header received
0926 03:38:07.442712 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.442723 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.442978 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.443105 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.445333 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.445342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.445356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.445376 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.445434 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.445454 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.445590 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.445795 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.445884 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7223,"thread_start_us":7063,"threads_started":1}
I20260926 03:38:07.447816 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.428681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35927 (local address 127.0.0.1:53846)
0926 03:38:07.444764 (+ 16083us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.444785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.444867 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.446463 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.446468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.446490 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.447005 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.447008 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.447024 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.447073 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.447531 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.447535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.447539 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.447607 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.447642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15995,"thread_start_us":13626,"threads_started":1}
I20260926 03:38:07.448539 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.429861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.1.221.130:35927)
0926 03:38:07.445316 (+ 15455us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.445324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.445354 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:38:07.445424 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.445431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.445647 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.445784 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.447233 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.447241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.447254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.447281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.447338 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.447358 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.448167 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.448220 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.448288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15259,"thread_start_us":15467,"threads_started":1}
I20260926 03:38:07.449491 6715 tablet_service.cc:1467] Processing CreateTablet for tablet fec67b510751477d85123aec4fd7f842 (DEFAULT_TABLE table=test-table [id=8e6c5642e9614ee893cc76573a40b48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:07.450066 6975 tablet_service.cc:1467] Processing CreateTablet for tablet fec67b510751477d85123aec4fd7f842 (DEFAULT_TABLE table=test-table [id=8e6c5642e9614ee893cc76573a40b48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:07.450973 6715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec67b510751477d85123aec4fd7f842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:07.451484 6975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec67b510751477d85123aec4fd7f842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:07.451476 6845 tablet_service.cc:1467] Processing CreateTablet for tablet fec67b510751477d85123aec4fd7f842 (DEFAULT_TABLE table=test-table [id=8e6c5642e9614ee893cc76573a40b48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:07.452497 6845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec67b510751477d85123aec4fd7f842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:07.462638 7058 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Bootstrap starting.
I20260926 03:38:07.465173 7059 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Bootstrap starting.
I20260926 03:38:07.466163 7060 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Bootstrap starting.
I20260926 03:38:07.467362 7058 tablet_bootstrap.cc:654] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:07.468678 7058 log.cc:824] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:07.468968 7059 tablet_bootstrap.cc:654] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:07.469522 7060 tablet_bootstrap.cc:654] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:07.470120 7059 log.cc:824] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:07.470537 7060 log.cc:824] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:07.471968 7058 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: No bootstrap required, opened a new log
I20260926 03:38:07.472249 7058 ts_tablet_manager.cc:1397] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260926 03:38:07.473084 7060 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: No bootstrap required, opened a new log
I20260926 03:38:07.473109 7059 tablet_bootstrap.cc:492] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: No bootstrap required, opened a new log
I20260926 03:38:07.473346 7059 ts_tablet_manager.cc:1397] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:38:07.473603 7060 ts_tablet_manager.cc:1397] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:38:07.480691 7060 raft_consensus.cc:348] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.481048 7060 raft_consensus.cc:374] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:07.481153 7060 raft_consensus.cc:729] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7596e55353f483ba6ef632c33506705, State: Initialized, Role: FOLLOWER
I20260926 03:38:07.481715 7060 consensus_queue.cc:260] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.482749 7040 heartbeater.cc:503] Master 127.1.221.190:41475 was elected leader, sending a full tablet report...
I20260926 03:38:07.482904 7058 raft_consensus.cc:348] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.483243 7060 ts_tablet_manager.cc:1428] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 03:38:07.483321 7058 raft_consensus.cc:374] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:07.483440 7058 raft_consensus.cc:729] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7b2ef47c97434c82f0c10855af92b4, State: Initialized, Role: FOLLOWER
I20260926 03:38:07.484040 7058 consensus_queue.cc:260] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.483994 7059 raft_consensus.cc:348] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.484398 7059 raft_consensus.cc:374] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:07.484488 7059 raft_consensus.cc:729] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da7504e33784cf5be16a31cf027642f, State: Initialized, Role: FOLLOWER
I20260926 03:38:07.485040 7059 consensus_queue.cc:260] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.486017 6780 heartbeater.cc:503] Master 127.1.221.190:41475 was elected leader, sending a full tablet report...
I20260926 03:38:07.486661 7059 ts_tablet_manager.cc:1428] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:38:07.487746 6910 heartbeater.cc:503] Master 127.1.221.190:41475 was elected leader, sending a full tablet report...
I20260926 03:38:07.488415 7058 ts_tablet_manager.cc:1428] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260926 03:38:07.498232 7066 raft_consensus.cc:482] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:07.498466 7066 raft_consensus.cc:504] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.498935 7064 raft_consensus.cc:482] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:07.499202 7064 raft_consensus.cc:504] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.500046 7066 leader_election.cc:288] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7596e55353f483ba6ef632c33506705 (127.1.221.131:39199), 4c7b2ef47c97434c82f0c10855af92b4 (127.1.221.130:35927)
I20260926 03:38:07.500938 7064 leader_election.cc:288] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da7504e33784cf5be16a31cf027642f (127.1.221.129:45413), 4c7b2ef47c97434c82f0c10855af92b4 (127.1.221.130:35927)
I20260926 03:38:07.503589 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.500257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:39199 (local address 127.1.221.129:59443)
0926 03:38:07.500962 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.500980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.501185 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.501696 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.501701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.501725 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.502442 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.502446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.502466 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.502520 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.502943 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.502948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.502953 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.503093 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.503150 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":1954,"threads_started":1}
I20260926 03:38:07.503907 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.500700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59443 (local address 127.1.221.131:39199)
0926 03:38:07.500987 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.500996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.501073 (+ 77us) server_negotiation.cc:407] Connection header received
0926 03:38:07.501332 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.501338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.501423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.501488 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.502686 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.502692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.502698 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.502718 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.502756 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.502772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.502862 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.503617 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.503676 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"spinlock_wait_cycles":9216}
I20260926 03:38:07.504451 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.501087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.1.221.131:51531)
0926 03:38:07.501297 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.501315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.501402 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.502986 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.502992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.503020 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.503594 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.503597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.503614 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.503669 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.504056 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.504061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.504066 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.504165 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.504218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"spinlock_wait_cycles":31872}
I20260926 03:38:07.504827 6995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "2da7504e33784cf5be16a31cf027642f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7596e55353f483ba6ef632c33506705" is_pre_election: true
I20260926 03:38:07.505281 6995 raft_consensus.cc:2454] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2da7504e33784cf5be16a31cf027642f in term 0.
I20260926 03:38:07.505476 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.502440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51531 (local address 127.1.221.129:45413)
0926 03:38:07.502599 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.502605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.502624 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:07.502673 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.502678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.502730 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.502825 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.503831 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.503837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.503842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.503860 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.503891 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.503905 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.504003 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.504750 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.504813 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:38:07.506121 6669 leader_election.cc:302] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da7504e33784cf5be16a31cf027642f, a7596e55353f483ba6ef632c33506705; no voters:
I20260926 03:38:07.506296 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.500419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:40467 (local address 127.1.221.130:35927)
0926 03:38:07.500626 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.500633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.504093 (+ 3460us) server_negotiation.cc:407] Connection header received
0926 03:38:07.504297 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.504302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.504355 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.504420 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.505445 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.505450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.505453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.505471 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.505503 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.505517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.505604 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.506003 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.506059 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:38:07.506573 7066 raft_consensus.cc:2781] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:07.506711 7066 raft_consensus.cc:482] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:07.506762 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.501576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35927 (local address 127.1.221.131:56395)
0926 03:38:07.504205 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.504220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.504353 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.504860 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.504864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.504886 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.505517 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.505520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.505535 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.505578 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.505974 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.505978 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.505982 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.506364 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.506412 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2498,"thread_start_us":3657,"threads_started":1}
I20260926 03:38:07.506820 7066 raft_consensus.cc:3037] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:07.506796 6735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "a7596e55353f483ba6ef632c33506705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da7504e33784cf5be16a31cf027642f" is_pre_election: true
I20260926 03:38:07.507496 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.501789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:56395 (local address 127.1.221.130:35927)
0926 03:38:07.503743 (+ 1954us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.503748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.504284 (+ 536us) server_negotiation.cc:407] Connection header received
0926 03:38:07.504471 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.504476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.504524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.504596 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.505729 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.505734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.505738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.505754 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.505796 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.505810 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.505891 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.506097 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.506164 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1847,"thread_start_us":1650,"threads_started":1}
I20260926 03:38:07.507510 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.500329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35927 (local address 127.1.221.129:40467)
0926 03:38:07.503945 (+ 3616us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.504020 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:07.504165 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:07.504655 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:07.504660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:07.504691 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:07.505234 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:07.505237 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:07.505252 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:07.505294 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:07.505873 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:07.505877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:07.505881 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:38:07.507214 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0926 03:38:07.507266 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3533,"thread_start_us":321,"threads_started":1}
I20260926 03:38:07.508371 6864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "2da7504e33784cf5be16a31cf027642f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7b2ef47c97434c82f0c10855af92b4" is_pre_election: true
I20260926 03:38:07.508430 6865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "a7596e55353f483ba6ef632c33506705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7b2ef47c97434c82f0c10855af92b4" is_pre_election: true
I20260926 03:38:07.508857 6864 raft_consensus.cc:2454] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2da7504e33784cf5be16a31cf027642f in term 0.
I20260926 03:38:07.509842 7066 raft_consensus.cc:504] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.510218 6735 raft_consensus.cc:2379] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a7596e55353f483ba6ef632c33506705 in current term 1: Already voted for candidate 2da7504e33784cf5be16a31cf027642f in this term.
I20260926 03:38:07.510411 7066 leader_election.cc:288] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [CANDIDATE]: Term 1 election: Requested vote from peers a7596e55353f483ba6ef632c33506705 (127.1.221.131:39199), 4c7b2ef47c97434c82f0c10855af92b4 (127.1.221.130:35927)
I20260926 03:38:07.510807 6995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "2da7504e33784cf5be16a31cf027642f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7596e55353f483ba6ef632c33506705"
I20260926 03:38:07.510900 6865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec67b510751477d85123aec4fd7f842" candidate_uuid: "2da7504e33784cf5be16a31cf027642f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7b2ef47c97434c82f0c10855af92b4"
I20260926 03:38:07.511013 6995 raft_consensus.cc:3037] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:07.511091 6865 raft_consensus.cc:3037] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:07.511847 6930 leader_election.cc:302] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7596e55353f483ba6ef632c33506705; no voters: 2da7504e33784cf5be16a31cf027642f, 4c7b2ef47c97434c82f0c10855af92b4
I20260926 03:38:07.513509 6865 raft_consensus.cc:2454] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da7504e33784cf5be16a31cf027642f in term 1.
I20260926 03:38:07.513576 6995 raft_consensus.cc:2454] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da7504e33784cf5be16a31cf027642f in term 1.
I20260926 03:38:07.513736 7064 raft_consensus.cc:2726] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:38:07.513847 6668 leader_election.cc:302] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da7504e33784cf5be16a31cf027642f, 4c7b2ef47c97434c82f0c10855af92b4; no voters:
I20260926 03:38:07.514117 7066 raft_consensus.cc:2781] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:07.514531 7066 raft_consensus.cc:686] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 LEADER]: Becoming Leader. State: Replica: 2da7504e33784cf5be16a31cf027642f, State: Running, Role: LEADER
I20260926 03:38:07.514971 7066 consensus_queue.cc:237] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } }
I20260926 03:38:07.518932 6594 catalog_manager.cc:5495] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2da7504e33784cf5be16a31cf027642f (127.1.221.129). New cstate: current_term: 1 leader_uuid: "2da7504e33784cf5be16a31cf027642f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da7504e33784cf5be16a31cf027642f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 } health_report { overall_health: UNKNOWN } } }
I20260926 03:38:07.534039 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.532310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.1.221.129:45413)
0926 03:38:07.532473 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.532477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.532494 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:38:07.532642 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.532646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.532730 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.532808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.533559 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.533563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.533566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.533578 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.533604 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.533614 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.533694 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.533755 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.533797 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:38:07.541513 6865 raft_consensus.cc:1261] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 1 FOLLOWER]: Refusing update from remote peer 2da7504e33784cf5be16a31cf027642f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:07.541611 6995 raft_consensus.cc:1261] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 1 FOLLOWER]: Refusing update from remote peer 2da7504e33784cf5be16a31cf027642f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:07.542327 7071 consensus_queue.cc:1035] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c7b2ef47c97434c82f0c10855af92b4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:07.542634 7066 consensus_queue.cc:1035] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7596e55353f483ba6ef632c33506705" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:07.570448 6593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:07.570722 6593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:07.574066 6593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5bc9594f2f741e8b4f569a79ca0b39e: Sending DeleteTablet for 3 replicas of tablet fec67b510751477d85123aec4fd7f842
I20260926 03:38:07.575059 6715 tablet_service.cc:1514] Processing DeleteTablet for tablet fec67b510751477d85123aec4fd7f842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:07 UTC) from {username='slave'} at 127.0.0.1:33046
I20260926 03:38:07.575735 6975 tablet_service.cc:1514] Processing DeleteTablet for tablet fec67b510751477d85123aec4fd7f842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:07 UTC) from {username='slave'} at 127.0.0.1:53336
I20260926 03:38:07.577260 6845 tablet_service.cc:1514] Processing DeleteTablet for tablet fec67b510751477d85123aec4fd7f842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:07 UTC) from {username='slave'} at 127.0.0.1:53846
I20260926 03:38:07.578171 7086 tablet_replica.cc:330] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: stopping tablet replica
I20260926 03:38:07.578703 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6653
I20260926 03:38:07.578684 7086 raft_consensus.cc:2229] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:07.579347 7086 raft_consensus.cc:2258] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:07.582641 7086 ts_tablet_manager.cc:1905] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:07.583783 7085 tablet_replica.cc:330] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: stopping tablet replica
I20260926 03:38:07.583927 7084 tablet_replica.cc:330] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: stopping tablet replica
I20260926 03:38:07.584257 7085 raft_consensus.cc:2229] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:07.584484 7084 raft_consensus.cc:2229] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:07.584779 7085 raft_consensus.cc:2258] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:07.585415 7084 raft_consensus.cc:2258] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:07.586943 7085 ts_tablet_manager.cc:1905] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:07.587414 7084 ts_tablet_manager.cc:1905] T fec67b510751477d85123aec4fd7f842 P 2da7504e33784cf5be16a31cf027642f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:07.587606 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.580082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.1.221.129:45413)
0926 03:38:07.580257 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.580263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.583819 (+ 3556us) server_negotiation.cc:407] Connection header received
0926 03:38:07.584088 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.584094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.584158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.584228 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.586705 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.586710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.586715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.586735 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.586773 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.586788 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.586884 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.587198 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.587260 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:38:07.588563 6775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:07.589741 7086 ts_tablet_manager.cc:1918] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:07.589884 7086 log.cc:1196] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/fec67b510751477d85123aec4fd7f842
I20260926 03:38:07.590299 7086 ts_tablet_manager.cc:1939] T fec67b510751477d85123aec4fd7f842 P 4c7b2ef47c97434c82f0c10855af92b4: Deleting consensus metadata
I20260926 03:38:07.591852 6578 catalog_manager.cc:4841] TS 4c7b2ef47c97434c82f0c10855af92b4 (127.1.221.130:35927): tablet fec67b510751477d85123aec4fd7f842 (table test-table [id=8e6c5642e9614ee893cc76573a40b48b]) successfully deleted
I20260926 03:38:07.594944 7085 ts_tablet_manager.cc:1918] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:07.595096 7085 log.cc:1196] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/fec67b510751477d85123aec4fd7f842
I20260926 03:38:07.595546 7085 ts_tablet_manager.cc:1939] T fec67b510751477d85123aec4fd7f842 P a7596e55353f483ba6ef632c33506705: Deleting consensus metadata
I20260926 03:38:07.596964 6579 catalog_manager.cc:4841] TS a7596e55353f483ba6ef632c33506705 (127.1.221.131:39199): tablet fec67b510751477d85123aec4fd7f842 (table test-table [id=8e6c5642e9614ee893cc76573a40b48b]) successfully deleted
I20260926 03:38:07.681268 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6653
I20260926 03:38:07.695851 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6783
W20260926 03:38:07.695814 6580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:45413 (error 108)
I20260926 03:38:07.697526 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.697137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33078)
0926 03:38:07.697284 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.697390 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:38:07.697718 6580 catalog_manager.cc:4584] TS 2da7504e33784cf5be16a31cf027642f (127.1.221.129:45413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fec67b510751477d85123aec4fd7f842: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
I20260926 03:38:07.698606 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.696617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.1.221.130:35927)
0926 03:38:07.696808 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.696814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.696862 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:38:07.697045 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.697048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.697129 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.697225 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.698100 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.698105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.698108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.698123 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.698156 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.698167 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.698255 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.698320 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.698364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:38:07.699206 6905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:07.751641 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.751065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33094)
0926 03:38:07.751309 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.751449 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:38:07.752599 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.752275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33108)
0926 03:38:07.752400 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.752477 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 03:38:07.785476 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6783
I20260926 03:38:07.798828 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6913
I20260926 03:38:07.801679 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.799551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.1.221.131:39199)
0926 03:38:07.799724 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:07.799730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:07.799774 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:38:07.799980 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:07.799986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:07.800067 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:07.800152 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:07.800936 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:07.800943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:07.800948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:07.800965 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:07.800996 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:07.801008 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:07.801118 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:07.801313 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 03:38:07.801378 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:38:07.802330 7035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:07.812994 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.812441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33120)
0926 03:38:07.812695 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.812807 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:38:07.814040 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.813645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33122)
0926 03:38:07.813803 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.813889 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:38:07.888204 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6913
I20260926 03:38:07.894618 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.894174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33128)
0926 03:38:07.894372 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.894477 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:38:07.895663 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:07.895239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45413 (local address 127.0.0.1:33138)
0926 03:38:07.895462 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:07.895533 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:38:07.901404 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 6563
I20260926 03:38:07.902510 6624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:08.005087 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 6563
2026-09-26T03:38:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2407 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 03:38:08.041378 1910 mini_oidc.cc:152] Starting JWKS server
I20260926 03:38:08.043958 1910 webserver.cc:460] Webserver started at https://127.0.0.1:38353/ using document root <none> and password file <none>
I20260926 03:38:08.044232 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:38:08.044836 1910 webserver.cc:460] Webserver started at http://127.0.0.1:38511/ using document root <none> and password file <none>
2026-09-26T03:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:08Z Disabled control of system clock
I20260926 03:38:08.069205 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35863
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38353/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/testchainca.cert with env {}
W20260926 03:38:08.380000 7098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:08.380303 7098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:08.380435 7098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:08.387724 7098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:08.387838 7098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:08.387900 7098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:08.387980 7098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:08.388037 7098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:38:08.388118 7098 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:38:08.388168 7098 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38353/jwks.json
I20260926 03:38:08.398284 7098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38353/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:08.399736 7098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:08.401046 7098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:08.412508 7104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:08.412578 7103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:08.412546 7106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:08.413247 7098 server_base.cc:1034] running on GCE node
I20260926 03:38:08.414000 7098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:08.415459 7098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:08.416663 7098 hybrid_clock.cc:648] HybridClock initialized: now 1790393888416612 us; error 44 us; skew 500 ppm
I20260926 03:38:08.417004 7098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:08.419124 7098 webserver.cc:460] Webserver started at http://127.1.221.190:39363/ using document root <none> and password file <none>
I20260926 03:38:08.419857 7098 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:08.419961 7098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:08.420245 7098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:08.423081 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "b808f3be02af4f248a9c0f6a7e233c68"
format_stamp: "Formatted at 2026-09-26 03:38:08 on dist-test-slave-lqrk"
server_key: "68fd7dcbddff0717c83c76348f765b5d"
server_key_iv: "f85b7ddaa28b765f397e160706f381c8"
server_key_version: "encryptionkey@0"
I20260926 03:38:08.423934 7098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b808f3be02af4f248a9c0f6a7e233c68"
format_stamp: "Formatted at 2026-09-26 03:38:08 on dist-test-slave-lqrk"
server_key: "68fd7dcbddff0717c83c76348f765b5d"
server_key_iv: "f85b7ddaa28b765f397e160706f381c8"
server_key_version: "encryptionkey@0"
I20260926 03:38:08.430831 7098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 03:38:08.435304 7112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:08.436997 7098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:38:08.437163 7098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "b808f3be02af4f248a9c0f6a7e233c68"
format_stamp: "Formatted at 2026-09-26 03:38:08 on dist-test-slave-lqrk"
server_key: "68fd7dcbddff0717c83c76348f765b5d"
server_key_iv: "f85b7ddaa28b765f397e160706f381c8"
server_key_version: "encryptionkey@0"
I20260926 03:38:08.437374 7098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:08.461069 7098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:08.461907 7098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:08.462199 7098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:08.478092 7098 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35863
I20260926 03:38:08.478120 7163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35863 every 8 connection(s)
I20260926 03:38:08.479346 7098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:08.482928 7164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:08.486970 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7098
I20260926 03:38:08.487164 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:08.487799 1910 external_mini_cluster.cc:1428] Setting key 42d757e1f7d52d3de2165c1ea55c7177
I20260926 03:38:08.494963 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68: Bootstrap starting.
I20260926 03:38:08.498696 7164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:08.500072 7164 log.cc:824] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:08.503333 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68: No bootstrap required, opened a new log
I20260926 03:38:08.503340 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:08.489697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.1.221.190:35863)
0926 03:38:08.490766 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:08.490780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:08.490838 (+ 58us) server_negotiation.cc:407] Connection header received
0926 03:38:08.493281 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:08.493310 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:08.493941 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:08.494385 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:08.494927 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.496889 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.498290 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.499344 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.499564 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:08.500739 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:08.500751 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:08.500770 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:08.501350 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:08.501448 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:08.501476 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:08.501572 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:08.501792 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:38:08.502109 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":284,"threads_started":1}
I20260926 03:38:08.507193 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:38:08.511698 7164 raft_consensus.cc:348] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } }
I20260926 03:38:08.512027 7164 raft_consensus.cc:374] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:08.512122 7164 raft_consensus.cc:729] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b808f3be02af4f248a9c0f6a7e233c68, State: Initialized, Role: FOLLOWER
I20260926 03:38:08.512642 7164 consensus_queue.cc:260] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } }
I20260926 03:38:08.512847 7164 raft_consensus.cc:388] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:08.512966 7164 raft_consensus.cc:482] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:08.513118 7164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:08.515467 7164 raft_consensus.cc:504] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } }
I20260926 03:38:08.515959 7164 leader_election.cc:302] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b808f3be02af4f248a9c0f6a7e233c68; no voters:
I20260926 03:38:08.516480 7164 leader_election.cc:288] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:08.516606 7169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:08.517256 7169 raft_consensus.cc:686] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [term 1 LEADER]: Becoming Leader. State: Replica: b808f3be02af4f248a9c0f6a7e233c68, State: Running, Role: LEADER
I20260926 03:38:08.517740 7169 consensus_queue.cc:237] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } }
I20260926 03:38:08.518364 7164 sys_catalog.cc:564] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:08.520516 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b808f3be02af4f248a9c0f6a7e233c68. Latest consensus state: current_term: 1 leader_uuid: "b808f3be02af4f248a9c0f6a7e233c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } } }
I20260926 03:38:08.520961 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:08.520635 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b808f3be02af4f248a9c0f6a7e233c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b808f3be02af4f248a9c0f6a7e233c68" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35863 } } }
I20260926 03:38:08.521078 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:08.521745 7178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:08.526849 7178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:08.536425 7178 catalog_manager.cc:1347] Generated new cluster ID: 661003beb7314128b06000912b427a39
I20260926 03:38:08.536559 7178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:08.553179 7178 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:08.555085 7178 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:08.564886 7178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b808f3be02af4f248a9c0f6a7e233c68: Generated new TSK 0
I20260926 03:38:08.565732 7178 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:38:08.608260 7194 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 03:38:08.608466 7194 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:38:08.609467 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:08.587140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.1.221.190:35863)
0926 03:38:08.587357 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:08.587365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:08.587860 (+ 495us) server_negotiation.cc:407] Connection header received
0926 03:38:08.588188 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:08.588192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:08.588258 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:08.588377 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:38:08.588981 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.589721 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.591840 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.592282 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.592376 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:08.593108 (+ 732us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:38:08.608898 (+ 15790us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:38:08.609204 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:38:08.609611 7167 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 03:38:08.611438 7193 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:08.587085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35863 (local address 127.0.0.1:46642)
0926 03:38:08.587752 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:08.587772 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:08.587966 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:08.588421 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:08.588427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:08.588440 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:38:08.588827 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:08.588839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.589854 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.589860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:08.591681 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:08.591698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:08.592451 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:08.592458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:08.592647 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:08.592734 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:38:08.609227 (+ 16493us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:38:08.609608 (+ 381us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:35863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":320,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.221.190:35863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 03:38:08.613685 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7098
I20260926 03:38:08.614785 7159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:08.707756 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7098
2026-09-26T03:38:08Z chronyd exiting
I20260926 03:38:08.756171 1910 test_util.cc:175] -----------------------------------------------
I20260926 03:38:08.756309 1910 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393848592298-1910-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:38:08.760011 1910 mini_oidc.cc:152] Starting JWKS server
I20260926 03:38:08.761783 1910 webserver.cc:460] Webserver started at https://127.0.0.1:44515/ using document root <none> and password file <none>
I20260926 03:38:08.761991 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:38:08.762651 1910 webserver.cc:460] Webserver started at http://127.0.0.1:34275/ using document root <none> and password file <none>
2026-09-26T03:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:08Z Disabled control of system clock
I20260926 03:38:08.787770 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46779
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:46779
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/testchainca.cert with env {}
W20260926 03:38:09.098658 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:09.098973 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:09.099068 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:09.106261 7201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:09.106374 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:09.106423 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:09.106477 7201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:09.106518 7201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:38:09.106576 7201 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:38:09.106614 7201 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44515/jwks.json
I20260926 03:38:09.116469 7201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:46779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:09.117679 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:09.118953 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:09.130097 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:09.130152 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:09.130113 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:09.130918 7201 server_base.cc:1034] running on GCE node
I20260926 03:38:09.131723 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:09.133091 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:09.134284 7201 hybrid_clock.cc:648] HybridClock initialized: now 1790393889134238 us; error 39 us; skew 500 ppm
I20260926 03:38:09.134613 7201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:09.136780 7201 webserver.cc:460] Webserver started at http://127.1.221.190:37583/ using document root <none> and password file <none>
I20260926 03:38:09.137421 7201 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:09.137526 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:09.137815 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:09.140585 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "c6e98ac108ff411f88027b52704a0beb"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "a68448b183cdd406dc024de0612f685c"
server_key_iv: "a44b4061c502e5eaf7bc2c8af9b52e9d"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.141304 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "c6e98ac108ff411f88027b52704a0beb"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "a68448b183cdd406dc024de0612f685c"
server_key_iv: "a44b4061c502e5eaf7bc2c8af9b52e9d"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.146060 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 03:38:09.149206 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:09.150498 7201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:09.150658 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "c6e98ac108ff411f88027b52704a0beb"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "a68448b183cdd406dc024de0612f685c"
server_key_iv: "a44b4061c502e5eaf7bc2c8af9b52e9d"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.150846 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:09.161749 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:09.162509 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:09.162770 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:09.179055 7201 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:46779
I20260926 03:38:09.179018 7266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:46779 every 8 connection(s)
I20260926 03:38:09.180339 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:09.183884 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7201
I20260926 03:38:09.184059 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:09.184115 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:09.184851 1910 external_mini_cluster.cc:1428] Setting key 8cae629ba9e7fe2cf62867ca4b054276
I20260926 03:38:09.198809 7267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb: Bootstrap starting.
I20260926 03:38:09.198879 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:09.186772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.1.221.190:46779)
0926 03:38:09.187890 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:09.187914 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:09.187965 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:38:09.189453 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:09.189480 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:09.190068 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:09.190493 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:09.191012 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.192954 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.194126 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.195103 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.195284 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:09.196266 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:09.196276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:09.196293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:09.196977 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:09.197068 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:09.197093 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:09.197189 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:09.197341 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:38:09.197651 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":373,"threads_started":1}
I20260926 03:38:09.202039 7267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:09.202543 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:38:09.203186 7267 log.cc:824] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:09.206050 7267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb: No bootstrap required, opened a new log
I20260926 03:38:09.212919 7267 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } }
I20260926 03:38:09.213272 7267 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:09.213366 7267 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e98ac108ff411f88027b52704a0beb, State: Initialized, Role: FOLLOWER
I20260926 03:38:09.213894 7267 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } }
I20260926 03:38:09.214111 7267 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:09.214229 7267 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:09.214370 7267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:09.216473 7267 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } }
I20260926 03:38:09.217013 7267 leader_election.cc:302] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6e98ac108ff411f88027b52704a0beb; no voters:
I20260926 03:38:09.217582 7267 leader_election.cc:288] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:09.217695 7272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:09.218327 7272 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [term 1 LEADER]: Becoming Leader. State: Replica: c6e98ac108ff411f88027b52704a0beb, State: Running, Role: LEADER
I20260926 03:38:09.218897 7272 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } }
I20260926 03:38:09.219684 7267 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:09.221578 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6e98ac108ff411f88027b52704a0beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } } }
I20260926 03:38:09.221982 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:09.223604 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6e98ac108ff411f88027b52704a0beb. Latest consensus state: current_term: 1 leader_uuid: "c6e98ac108ff411f88027b52704a0beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e98ac108ff411f88027b52704a0beb" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46779 } } }
I20260926 03:38:09.223851 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:09.224354 7279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:09.229497 7279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:09.238910 7279 catalog_manager.cc:1347] Generated new cluster ID: 30ef44297e254014aa2ac8b11b7363ec
I20260926 03:38:09.239032 7279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:09.261348 7279 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:09.262620 7279 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:09.270403 7279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6e98ac108ff411f88027b52704a0beb: Generated new TSK 0
I20260926 03:38:09.271202 7279 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:38:09.310712 7297 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:38:09.310827 7297 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:38:09.312300 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:09.292483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.1.221.190:46779)
0926 03:38:09.292676 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:09.292682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:09.292976 (+ 294us) server_negotiation.cc:407] Connection header received
0926 03:38:09.293186 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:09.293192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:09.293278 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:09.293449 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:38:09.293846 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.294964 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.296290 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.296989 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.297129 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:09.297533 (+ 404us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:38:09.311548 (+ 14015us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:38:09.311965 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:38:09.312326 7296 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:09.292341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:46779 (local address 127.0.0.1:43876)
0926 03:38:09.292883 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:09.292899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:09.293021 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:09.293446 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:09.293449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:09.293457 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:38:09.293698 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:09.293706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.295085 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.295088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:09.296157 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:09.296169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.297127 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.297131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:09.297253 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:09.297284 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:38:09.311812 (+ 14528us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:38:09.312033 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:46779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":284,"threads_started":1}
W20260926 03:38:09.312438 7270 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 03:38:09.313468 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7201
I20260926 03:38:09.314788 7262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:09.406785 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7201
2026-09-26T03:38:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:38:09.458495 1910 mini_oidc.cc:152] Starting JWKS server
I20260926 03:38:09.460355 1910 webserver.cc:460] Webserver started at https://127.0.0.1:45035/ using document root <none> and password file <none>
I20260926 03:38:09.460603 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:38:09.461211 1910 webserver.cc:460] Webserver started at http://127.0.0.1:39995/ using document root <none> and password file <none>
2026-09-26T03:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:09Z Disabled control of system clock
I20260926 03:38:09.485380 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39227
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39227
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:38:09.808709 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:09.809008 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:09.809137 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:09.816358 7304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:09.816470 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:09.816540 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:09.816617 7304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:09.816676 7304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:38:09.816763 7304 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:38:09.816811 7304 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45035/jwks.json
I20260926 03:38:09.826339 7304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:09.827775 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:09.829587 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:09.841176 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:09.841202 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:09.841132 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:09.841969 7304 server_base.cc:1034] running on GCE node
I20260926 03:38:09.842715 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:09.844156 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:09.845350 7304 hybrid_clock.cc:648] HybridClock initialized: now 1790393889845300 us; error 42 us; skew 500 ppm
I20260926 03:38:09.845650 7304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:09.851788 7304 webserver.cc:460] Webserver started at http://127.1.221.190:42671/ using document root <none> and password file <none>
I20260926 03:38:09.852466 7304 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:09.852553 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:09.852818 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:09.855715 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "b0c77f1beb6148a39fa17e0820d8558f"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "3cea75cd9d4bac6a640ccb60a50406da"
server_key_iv: "62f7ccfca7e3991013a1fda1bfad5b5a"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.856442 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b0c77f1beb6148a39fa17e0820d8558f"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "3cea75cd9d4bac6a640ccb60a50406da"
server_key_iv: "62f7ccfca7e3991013a1fda1bfad5b5a"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.861438 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:38:09.864899 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:09.866287 7304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:38:09.866446 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "b0c77f1beb6148a39fa17e0820d8558f"
format_stamp: "Formatted at 2026-09-26 03:38:09 on dist-test-slave-lqrk"
server_key: "3cea75cd9d4bac6a640ccb60a50406da"
server_key_iv: "62f7ccfca7e3991013a1fda1bfad5b5a"
server_key_version: "encryptionkey@0"
I20260926 03:38:09.866649 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:09.904670 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:09.905563 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:09.905859 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:09.922662 7304 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39227
I20260926 03:38:09.922669 7369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39227 every 8 connection(s)
I20260926 03:38:09.924052 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:09.927677 7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:09.931321 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7304
I20260926 03:38:09.931546 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:09.932148 1910 external_mini_cluster.cc:1428] Setting key 16c05fe7b76186404e26e14a8f2e2cf0
I20260926 03:38:09.938706 7370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f: Bootstrap starting.
I20260926 03:38:09.942236 7370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:09.943581 7370 log.cc:824] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:09.946424 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:09.934014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.1.221.190:39227)
0926 03:38:09.934962 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:09.934974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:09.935022 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:38:09.937127 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:09.937150 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:09.937623 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:09.937947 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:09.938532 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.939899 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.941698 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:09.942378 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:09.942527 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:09.943680 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:09.943687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:09.943703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:09.944384 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:09.944501 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:09.944528 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:09.944655 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:09.944834 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:38:09.945185 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
I20260926 03:38:09.947008 7370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f: No bootstrap required, opened a new log
I20260926 03:38:09.950416 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:38:09.956341 7370 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } }
I20260926 03:38:09.956712 7370 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:09.956786 7370 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c77f1beb6148a39fa17e0820d8558f, State: Initialized, Role: FOLLOWER
I20260926 03:38:09.957330 7370 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } }
I20260926 03:38:09.957546 7370 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:09.957643 7370 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:09.957780 7370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:09.960335 7370 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } }
I20260926 03:38:09.960803 7370 leader_election.cc:302] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c77f1beb6148a39fa17e0820d8558f; no voters:
I20260926 03:38:09.961333 7370 leader_election.cc:288] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:09.961450 7375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:09.962347 7375 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [term 1 LEADER]: Becoming Leader. State: Replica: b0c77f1beb6148a39fa17e0820d8558f, State: Running, Role: LEADER
I20260926 03:38:09.962852 7375 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } }
I20260926 03:38:09.963258 7370 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:09.965706 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c77f1beb6148a39fa17e0820d8558f. Latest consensus state: current_term: 1 leader_uuid: "b0c77f1beb6148a39fa17e0820d8558f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } } }
I20260926 03:38:09.965875 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c77f1beb6148a39fa17e0820d8558f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c77f1beb6148a39fa17e0820d8558f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39227 } } }
I20260926 03:38:09.966164 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:09.966225 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:09.968842 7381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:09.974617 7381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:09.983239 7381 catalog_manager.cc:1347] Generated new cluster ID: 15131cc7d7e9462580b3dfc72b313cac
I20260926 03:38:09.983354 7381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:10.007629 7381 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:10.008967 7381 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:10.032840 7381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c77f1beb6148a39fa17e0820d8558f: Generated new TSK 0
I20260926 03:38:10.033761 7381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:10.061861 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:10.053189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.1.221.190:39227)
0926 03:38:10.053419 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:10.053426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:10.053830 (+ 404us) server_negotiation.cc:407] Connection header received
0926 03:38:10.054049 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:10.054052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:10.054125 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:10.054254 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:38:10.054765 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.055533 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.057548 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.058051 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.058163 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:10.058765 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:38:10.061140 (+ 2375us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:38:10.061490 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 03:38:10.061986 7373 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 03:38:10.063958 7399 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:10.053140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39227 (local address 127.0.0.1:47646)
0926 03:38:10.053723 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:10.053742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:10.053895 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:10.054287 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:10.054293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:10.054307 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:38:10.054623 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:10.054634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.055668 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.055673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:10.057358 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:10.057384 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.058243 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.058249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:10.058446 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:10.058494 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:38:10.061398 (+ 2904us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:38:10.063541 (+ 2143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:39227: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":1773,"thread_start_us":290,"threads_started":1}
I20260926 03:38:10.065703 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7304
I20260926 03:38:10.066967 7365 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 03:38:10.073768 7400 webserver.cc:533] Webserver: sslize failed:
E20260926 03:38:10.073884 7400 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:38:10.161084 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7304
2026-09-26T03:38:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (753 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:10Z Disabled control of system clock
I20260926 03:38:10.237517 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35233
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35233
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/testchainca.cert with env {}
W20260926 03:38:10.556656 7405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:10.556963 7405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:10.557055 7405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:10.564209 7405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:10.564335 7405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:10.564404 7405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:10.564484 7405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:10.564535 7405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:38:10.564594 7405 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:38:10.564631 7405 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:38:10.574396 7405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:10.575701 7405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:10.576929 7405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:10.588112 7410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:10.588040 7411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:10.588070 7413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:10.589280 7405 server_base.cc:1034] running on GCE node
I20260926 03:38:10.590068 7405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:10.591531 7405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:10.592742 7405 hybrid_clock.cc:648] HybridClock initialized: now 1790393890592682 us; error 54 us; skew 500 ppm
I20260926 03:38:10.593185 7405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:10.595605 7405 webserver.cc:460] Webserver started at http://127.1.221.190:34713/ using document root <none> and password file <none>
I20260926 03:38:10.596287 7405 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:10.596383 7405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:10.596676 7405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:10.599601 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "3c40c312c35d45dba43c038256ca5fb3"
format_stamp: "Formatted at 2026-09-26 03:38:10 on dist-test-slave-lqrk"
server_key: "04097d9866a49d1aae8d3bc556298c38"
server_key_iv: "66dc53a4d87e2f686aa896763db1d61c"
server_key_version: "encryptionkey@0"
I20260926 03:38:10.600392 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "3c40c312c35d45dba43c038256ca5fb3"
format_stamp: "Formatted at 2026-09-26 03:38:10 on dist-test-slave-lqrk"
server_key: "04097d9866a49d1aae8d3bc556298c38"
server_key_iv: "66dc53a4d87e2f686aa896763db1d61c"
server_key_version: "encryptionkey@0"
I20260926 03:38:10.605435 7405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:38:10.608829 7419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:10.610224 7405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:38:10.610376 7405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "3c40c312c35d45dba43c038256ca5fb3"
format_stamp: "Formatted at 2026-09-26 03:38:10 on dist-test-slave-lqrk"
server_key: "04097d9866a49d1aae8d3bc556298c38"
server_key_iv: "66dc53a4d87e2f686aa896763db1d61c"
server_key_version: "encryptionkey@0"
I20260926 03:38:10.610577 7405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:10.640379 7405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:10.641227 7405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:10.641482 7405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:10.658244 7405 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35233
I20260926 03:38:10.658264 7470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35233 every 8 connection(s)
I20260926 03:38:10.659677 7405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:10.663429 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:10.666612 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7405
I20260926 03:38:10.666818 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:10.667496 1910 external_mini_cluster.cc:1428] Setting key 2e2357b24c8eb73084a711ef7c03a612
I20260926 03:38:10.676916 7471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3: Bootstrap starting.
I20260926 03:38:10.680640 7471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:10.681945 7471 log.cc:824] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:10.682431 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:10.669425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.1.221.190:35233)
0926 03:38:10.670455 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:10.670470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:10.670525 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:38:10.672466 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:10.672495 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:10.673141 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:10.673581 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:10.674131 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.676074 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.677398 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.678380 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.678603 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:10.679625 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:10.679637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:10.679657 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:10.680349 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:10.680462 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:10.680489 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:10.680610 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:10.680789 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:38:10.681163 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":294,"threads_started":1}
I20260926 03:38:10.684938 7471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3: No bootstrap required, opened a new log
I20260926 03:38:10.686367 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:38:10.693251 7471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } }
I20260926 03:38:10.693614 7471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:10.693684 7471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c40c312c35d45dba43c038256ca5fb3, State: Initialized, Role: FOLLOWER
I20260926 03:38:10.694188 7471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } }
I20260926 03:38:10.694383 7471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:10.694476 7471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:10.694586 7471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:10.696695 7471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } }
I20260926 03:38:10.697175 7471 leader_election.cc:302] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c40c312c35d45dba43c038256ca5fb3; no voters:
I20260926 03:38:10.697718 7471 leader_election.cc:288] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:10.697829 7476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:10.698485 7476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c40c312c35d45dba43c038256ca5fb3, State: Running, Role: LEADER
I20260926 03:38:10.699016 7476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } }
I20260926 03:38:10.699828 7471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:10.701521 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c40c312c35d45dba43c038256ca5fb3. Latest consensus state: current_term: 1 leader_uuid: "3c40c312c35d45dba43c038256ca5fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } } }
I20260926 03:38:10.701858 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:10.701735 7477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c40c312c35d45dba43c038256ca5fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c40c312c35d45dba43c038256ca5fb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35233 } } }
I20260926 03:38:10.702118 7477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:10.704021 7483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:10.709815 7483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:10.719933 7483 catalog_manager.cc:1347] Generated new cluster ID: 2047edad0d0c4897b79f63c91be5ef3e
I20260926 03:38:10.720042 7483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:10.750753 7483 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:10.752370 7483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:10.758244 7483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c40c312c35d45dba43c038256ca5fb3: Generated new TSK 0
I20260926 03:38:10.759022 7483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:10.852205 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:10.777878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.1.221.190:35233)
0926 03:38:10.778083 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:10.778090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:10.778345 (+ 255us) server_negotiation.cc:407] Connection header received
0926 03:38:10.778559 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:10.778565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:10.778659 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:10.778832 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:38:10.779235 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.780402 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.781774 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.782541 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.782666 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:10.783206 (+ 540us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:38:10.851473 (+ 68267us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:38:10.851914 (+ 441us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41042: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:38:10.852332 7500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:10.777698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35233 (local address 127.0.0.1:41042)
0926 03:38:10.778247 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:10.778264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:10.778387 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:10.778831 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:10.778834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:10.778843 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:38:10.779091 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:10.779099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.780541 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.780545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:10.781631 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:10.781643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:10.782757 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:10.782763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:10.782909 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:10.782947 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:38:10.851773 (+ 68826us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:38:10.852017 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:35233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
W20260926 03:38:10.852353 7474 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41042: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 03:38:10.853788 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7405
I20260926 03:38:10.854930 7466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:10.954429 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7405
2026-09-26T03:38:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (773 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1373 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1370 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:13 dist-test-slave-lqrk krb5kdc[7517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:13 dist-test-slave-lqrk krb5kdc[7517](info): set up 2 sockets
Sep 26 03:38:13 dist-test-slave-lqrk krb5kdc[7517](info): commencing operation
krb5kdc: starting...
W20260926 03:38:14.687947 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:14 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:38:14.884931 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39045
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:15.199702 7530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:15.200047 7530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:15.200171 7530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:15.208958 7530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:15.209091 7530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:15.209170 7530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:15.209256 7530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:15.209316 7530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:15.219413 7530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:15.220837 7530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:15.222141 7530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:15.233891 7535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:15.233969 7538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:15.233945 7536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:15.236974 7530 server_base.cc:1034] running on GCE node
I20260926 03:38:15.237864 7530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:15.239497 7530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:15.240720 7530 hybrid_clock.cc:648] HybridClock initialized: now 1790393895240666 us; error 49 us; skew 500 ppm
I20260926 03:38:15.241072 7530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:15 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393895, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:15.248557 7530 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:38:15.250269 7530 webserver.cc:460] Webserver started at http://127.1.221.190:39369/ using document root <none> and password file <none>
I20260926 03:38:15.251060 7530 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:15.251189 7530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:15.251566 7530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:15.254730 7530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "648efaea7e947be99f6ea3be0a5180c1"
server_key_iv: "9242ff80ec428b02a373ea3ea85fa671"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.255568 7530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "648efaea7e947be99f6ea3be0a5180c1"
server_key_iv: "9242ff80ec428b02a373ea3ea85fa671"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.260916 7530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:38:15.264590 7545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:15.265980 7530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:15.266144 7530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "648efaea7e947be99f6ea3be0a5180c1"
server_key_iv: "9242ff80ec428b02a373ea3ea85fa671"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.266332 7530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:15.281414 7530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:15.285516 7530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:15.285768 7530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:15.302979 7530 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39045
I20260926 03:38:15.302997 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39045 every 8 connection(s)
I20260926 03:38:15.304407 7530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:15.308193 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:15.314186 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7530
I20260926 03:38:15.314400 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:15.315183 1910 external_mini_cluster.cc:1428] Setting key 4ea4d0c054be51c3b5448994207baaeb
I20260926 03:38:15.319869 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3: Bootstrap starting.
I20260926 03:38:15.323621 7597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:15.324760 7597 log.cc:824] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:15.327953 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3: No bootstrap required, opened a new log
Sep 26 03:38:15 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:15.336881 7597 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } }
I20260926 03:38:15.337402 7597 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:15.337519 7597 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1f8b10f65e492fa1edd6d50bcdceb3, State: Initialized, Role: FOLLOWER
I20260926 03:38:15.338204 7597 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } }
I20260926 03:38:15.338451 7597 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:15.338587 7597 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:15.338745 7597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:15.339051 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:15.317466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.1.221.190:39045)
0926 03:38:15.318315 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:15.318328 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:15.318379 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:38:15.320225 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:15.320251 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:15.320691 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:15.321027 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:15.322787 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.324166 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.325780 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.326537 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.326665 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:15.331330 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:15.331341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:15.331361 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:15.331440 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:15.334543 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:15.335480 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:15.335487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:15.335495 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:15.335686 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:15.336093 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:15.336097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:15.336099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:15.336667 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:15.336907 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:15.337367 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 03:38:15.337821 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":400,"threads_started":1}
I20260926 03:38:15.341162 7597 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } }
I20260926 03:38:15.341665 7597 leader_election.cc:302] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1f8b10f65e492fa1edd6d50bcdceb3; no voters:
I20260926 03:38:15.342267 7597 leader_election.cc:288] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:15.342373 7602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:15.342968 7602 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [term 1 LEADER]: Becoming Leader. State: Replica: fe1f8b10f65e492fa1edd6d50bcdceb3, State: Running, Role: LEADER
I20260926 03:38:15.343538 7602 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } }
I20260926 03:38:15.344223 7597 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:15.346503 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1f8b10f65e492fa1edd6d50bcdceb3. Latest consensus state: current_term: 1 leader_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } } }
I20260926 03:38:15.346582 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1f8b10f65e492fa1edd6d50bcdceb3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39045 } } }
I20260926 03:38:15.346937 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:15.346942 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:15.348672 7607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:15.353305 7607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:15.361852 7607 catalog_manager.cc:1347] Generated new cluster ID: f7ec4c2791e547cea08330c0b83cff06
I20260926 03:38:15.361974 7607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:15.373862 7607 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:15.375844 7607 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:15.391355 7607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1f8b10f65e492fa1edd6d50bcdceb3: Generated new TSK 0
I20260926 03:38:15.392583 7607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:38:15.447227 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:15.769603 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:15.769905 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:15.770040 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:15.777379 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:15.777498 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:15.777645 7623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:15.787418 7623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:15.788725 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:15.790035 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:15.803524 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:15.803557 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:15.803516 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:15.804872 7623 server_base.cc:1034] running on GCE node
I20260926 03:38:15.805500 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:15.806497 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:15.807729 7623 hybrid_clock.cc:648] HybridClock initialized: now 1790393895807643 us; error 57 us; skew 500 ppm
I20260926 03:38:15.808084 7623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:15 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393895, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:15.811463 7623 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:38:15.813143 7623 webserver.cc:460] Webserver started at http://127.1.221.129:44175/ using document root <none> and password file <none>
I20260926 03:38:15.813853 7623 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:15.813977 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:15.814294 7623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:15.817385 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "6c15098c4be147a1bdee0ed61486e5ea"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "b21fb3feb36b2204f4cf264eab049fa7"
server_key_iv: "f797fef7ff62672cbd77c3c17a2f971b"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.818270 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "6c15098c4be147a1bdee0ed61486e5ea"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "b21fb3feb36b2204f4cf264eab049fa7"
server_key_iv: "f797fef7ff62672cbd77c3c17a2f971b"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.823890 7623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:15.827548 7638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:15.828902 7623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:15.829071 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "6c15098c4be147a1bdee0ed61486e5ea"
format_stamp: "Formatted at 2026-09-26 03:38:15 on dist-test-slave-lqrk"
server_key: "b21fb3feb36b2204f4cf264eab049fa7"
server_key_iv: "f797fef7ff62672cbd77c3c17a2f971b"
server_key_version: "encryptionkey@0"
I20260926 03:38:15.829303 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:15.843632 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:15.848084 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:15.848342 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:15.849318 7623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:15.850659 7623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:15.850759 7623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:15.850852 7623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:15.850895 7623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:15.882006 7623 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41279
I20260926 03:38:15.882045 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41279 every 8 connection(s)
I20260926 03:38:15.883287 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:15.889256 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7623
I20260926 03:38:15.889461 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:15.890213 1910 external_mini_cluster.cc:1428] Setting key 983599d49941082edee50c64812eb58d
Sep 26 03:38:15 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393895, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:15.905304 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:15.885813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59977 (local address 127.1.221.190:39045)
0926 03:38:15.886230 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:15.886235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:15.887249 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 03:38:15.889169 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:15.889174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:15.889248 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:15.889335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:15.891362 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.892457 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.894753 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.895159 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.895258 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:15.899317 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:15.899323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:15.899326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:15.899363 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:15.901915 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:15.902735 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:15.902742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:15.902746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:15.902893 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:15.903358 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:15.903365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:15.903367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:15.903703 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:15.903854 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:15.904619 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 03:38:15.904870 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260926 03:38:15.906456 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:15.886375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39045 (local address 127.1.221.129:59977)
0926 03:38:15.887076 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:15.887119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:15.888818 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:15.889657 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:15.889670 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:15.890418 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:15.891207 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:15.891228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.892615 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.892621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:15.894588 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:15.894604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:15.895399 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:15.895406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:15.895672 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:15.896386 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:15.896414 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:15.899070 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:15.902104 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:15.902115 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:15.902134 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:15.902599 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:15.903009 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:15.903014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:15.903019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:15.903239 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:15.903841 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:15.903851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:15.904222 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 03:38:15.904500 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 03:38:15.905094 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
I20260926 03:38:15.908249 7751 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39045
I20260926 03:38:15.908561 7751 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:15.909286 7751 heartbeater.cc:511] Master 127.1.221.190:39045 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:38:15.911419 7562 ts_manager.cc:194] Registered new tserver with Master: 6c15098c4be147a1bdee0ed61486e5ea (127.1.221.129:41279)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:38:15.913048 7562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:59977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:38:15.935045 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:16.252637 7757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:16.252912 7757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:16.253043 7757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:16.260143 7757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:16.260253 7757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:16.260408 7757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:16.270219 7757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:16.271538 7757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:16.272857 7757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:16.285143 7762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:16.285176 7765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:16.285233 7763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:16.286335 7757 server_base.cc:1034] running on GCE node
I20260926 03:38:16.286922 7757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:16.287849 7757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:16.289040 7757 hybrid_clock.cc:648] HybridClock initialized: now 1790393896288997 us; error 38 us; skew 500 ppm
I20260926 03:38:16.289326 7757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.292210 7757 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:38:16.293546 7757 webserver.cc:460] Webserver started at http://127.1.221.130:33605/ using document root <none> and password file <none>
I20260926 03:38:16.294157 7757 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:16.294248 7757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:16.294503 7757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:16.297334 7757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "14cbfe63549c4eaea547d4ab4a3d7acf"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "9f9fd768d65adbfd2295b1dce65265d7"
server_key_iv: "d049f30d3110d0ce1be256d502a33883"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.298048 7757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "14cbfe63549c4eaea547d4ab4a3d7acf"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "9f9fd768d65adbfd2295b1dce65265d7"
server_key_iv: "d049f30d3110d0ce1be256d502a33883"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.303278 7757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:38:16.307449 7772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.308969 7757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:16.309121 7757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "14cbfe63549c4eaea547d4ab4a3d7acf"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "9f9fd768d65adbfd2295b1dce65265d7"
server_key_iv: "d049f30d3110d0ce1be256d502a33883"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.309326 7757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:16.326895 7757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:16.330639 7757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:16.330871 7757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:16.331799 7757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:16.333166 7757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:16.333238 7757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.333338 7757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:16.333400 7757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.361814 7757 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45227
I20260926 03:38:16.361871 7884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45227 every 8 connection(s)
I20260926 03:38:16.363143 7757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:16.363643 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7757
I20260926 03:38:16.363817 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:16.364382 1910 external_mini_cluster.cc:1428] Setting key b5b5fd42fc70f1d708bf9bf6cc784ffd
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:16.383936 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.365990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:50251 (local address 127.1.221.190:39045)
0926 03:38:16.366183 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.366190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.367522 (+ 1332us) server_negotiation.cc:407] Connection header received
0926 03:38:16.369364 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.369370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.369456 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.369542 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.371097 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.371849 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.373340 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.373910 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.374029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.377425 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.377432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.377436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.377475 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.380750 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.381492 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.381498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.381501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.381660 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.382073 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.382078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.382081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.382459 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.382612 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.383303 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.383579 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:38:16.384791 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.366550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39045 (local address 127.1.221.130:50251)
0926 03:38:16.367349 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:16.367418 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:16.369077 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:16.369799 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:16.369807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:16.370355 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:16.370927 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:16.370947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.371991 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.371995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:16.373202 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:16.373215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.374126 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.374131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:16.374353 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.374979 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:16.375001 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:16.377168 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:16.380905 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:16.380912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:16.380928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:16.381370 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:16.381760 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:16.381763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:16.381765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:16.381972 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:16.382582 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:16.382588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:16.382954 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 03:38:16.383187 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:38:16.383632 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":334,"threads_started":1}
I20260926 03:38:16.386204 7885 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39045
I20260926 03:38:16.386499 7885 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:38:16.387373 7885 heartbeater.cc:511] Master 127.1.221.190:39045 requested a full tablet report, sending...
I20260926 03:38:16.388952 7562 ts_manager.cc:194] Registered new tserver with Master: 14cbfe63549c4eaea547d4ab4a3d7acf (127.1.221.130:45227)
I20260926 03:38:16.389739 7562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:50251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:38:16.411132 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:16.722237 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:16.722541 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:16.722661 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:16.729826 7890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:16.729926 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:16.730072 7890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:16.739719 7890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:16.740911 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:16.742153 7890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:16.754572 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:16.754587 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:16.754825 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:16.755167 7890 server_base.cc:1034] running on GCE node
I20260926 03:38:16.755795 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:16.756639 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:16.757836 7890 hybrid_clock.cc:648] HybridClock initialized: now 1790393896757781 us; error 42 us; skew 500 ppm
I20260926 03:38:16.758128 7890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.760952 7890 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:38:16.762223 7890 webserver.cc:460] Webserver started at http://127.1.221.131:39551/ using document root <none> and password file <none>
I20260926 03:38:16.762853 7890 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:16.762954 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:16.763231 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:16.765996 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "dc8c6d85d92342f38bcabc9b7ac60834"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "fa29bbb975b19f0560ac68adfb064747"
server_key_iv: "f5da0d83c3d79f01280f412191bad307"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.766700 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "dc8c6d85d92342f38bcabc9b7ac60834"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "fa29bbb975b19f0560ac68adfb064747"
server_key_iv: "f5da0d83c3d79f01280f412191bad307"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.771449 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 03:38:16.774644 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.776026 7890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:16.776163 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "dc8c6d85d92342f38bcabc9b7ac60834"
format_stamp: "Formatted at 2026-09-26 03:38:16 on dist-test-slave-lqrk"
server_key: "fa29bbb975b19f0560ac68adfb064747"
server_key_iv: "f5da0d83c3d79f01280f412191bad307"
server_key_version: "encryptionkey@0"
I20260926 03:38:16.776335 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:16.795094 7890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:16.798928 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:16.799180 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:16.800133 7890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:16.801436 7890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:16.801509 7890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.801623 7890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:16.801673 7890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:16.830686 7890 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:33051
I20260926 03:38:16.830731 8017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:33051 every 8 connection(s)
I20260926 03:38:16.831995 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:16.839807 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 7890
I20260926 03:38:16.839967 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:16.840485 1910 external_mini_cluster.cc:1428] Setting key d00391935f9bb52f4a864287d12c6d6d
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:16.850380 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.834553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54809 (local address 127.1.221.190:39045)
0926 03:38:16.834758 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.834765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.835935 (+ 1170us) server_negotiation.cc:407] Connection header received
0926 03:38:16.837623 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.837627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.837695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.837778 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.839160 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.840007 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.841911 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.842360 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.842461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.845704 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.845709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.845713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.845745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.847992 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.848598 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.848601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.848604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.848703 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.849047 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.849050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.849052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.849302 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.849401 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.849988 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.850148 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:38:16.851410 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.835054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39045 (local address 127.1.221.131:54809)
0926 03:38:16.835789 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:16.835831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:16.837394 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:16.837968 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:16.837975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:16.838469 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:16.839029 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:16.839045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.840149 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.840154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:16.841789 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:16.841803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.842549 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.842554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:16.842737 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.843358 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:16.843418 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:16.845542 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:16.848105 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:16.848112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:16.848124 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:16.848508 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:16.848774 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:16.848777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:16.848779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:16.848950 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:16.849396 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:16.849402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:16.849695 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 03:38:16.849928 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:38:16.850292 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":312,"threads_started":1}
I20260926 03:38:16.852648 8018 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39045
I20260926 03:38:16.852949 8018 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:16.853724 8018 heartbeater.cc:511] Master 127.1.221.190:39045 requested a full tablet report, sending...
I20260926 03:38:16.855109 7562 ts_manager.cc:194] Registered new tserver with Master: dc8c6d85d92342f38bcabc9b7ac60834 (127.1.221.131:33051)
I20260926 03:38:16.855649 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:38:16.855985 7562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:54809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/test-admin.keytab.
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.880939 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:38:16.886878 8025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:16.883882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.1.221.129:41279)
0926 03:38:16.884604 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.884613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.885011 (+ 398us) server_negotiation.cc:407] Connection header received
0926 03:38:16.885219 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.885228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.885487 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.885615 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.886489 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51404: BlockingRecv error: recv got EOF from 127.0.0.1:51404 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":357,"threads_started":1}
W20260926 03:38:16.886988 1910 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.888600 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.889998 1910 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:16.904929 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.891595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.1.221.129:41279)
0926 03:38:16.891762 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.891768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.891942 (+ 174us) server_negotiation.cc:407] Connection header received
0926 03:38:16.892116 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.892120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.892179 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.892273 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.893166 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.894131 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.895241 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.895848 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.895967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.899483 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.899496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.899510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.899546 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.902126 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.902834 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.902843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.902845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.902968 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.903333 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.903338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.903340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.903729 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.904012 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.904372 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.904677 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 03:38:16.910347 8025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:16.908787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.1.221.129:41279)
0926 03:38:16.908930 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.908934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.909199 (+ 265us) server_negotiation.cc:407] Connection header received
0926 03:38:16.909325 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.909328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.909393 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.909469 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.910156 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51418: BlockingRecv error: recv got EOF from 127.0.0.1:51418 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 03:38:16.910583 1910 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.912353 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.913585 1910 init.cc:295] Successfully reacquired a new kerberos TGT
I20260926 03:38:16.916437 7751 heartbeater.cc:503] Master 127.1.221.190:39045 was elected leader, sending a full tablet report...
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:16.926949 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.914808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.1.221.129:41279)
0926 03:38:16.914958 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.914964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.915295 (+ 331us) server_negotiation.cc:407] Connection header received
0926 03:38:16.915465 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.915470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.915525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.915603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.916496 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.917331 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.918547 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.919006 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.919110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.922302 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.922308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.922311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.922341 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.924711 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.925279 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.925285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.925288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.925410 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.925743 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.925748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.925752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.926009 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.926243 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.926498 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.926685 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 03:38:16.930662 8025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:16.929293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.1.221.129:41279)
0926 03:38:16.929496 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.929501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.929660 (+ 159us) server_negotiation.cc:407] Connection header received
0926 03:38:16.929775 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.929778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.929844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.929921 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.930447 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51450: BlockingRecv error: recv got EOF from 127.0.0.1:51450 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 03:38:16.930816 1910 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393848592298-1910-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.932245 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:16.933472 1910 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:38:16 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:16.946269 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.934664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.1.221.129:41279)
0926 03:38:16.934900 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.934905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.935104 (+ 199us) server_negotiation.cc:407] Connection header received
0926 03:38:16.935211 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.935214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.935269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.935341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.936227 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.936939 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.938082 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.938493 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.938585 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.941712 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.941717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.941720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.941755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.944174 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.944700 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.944703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.944706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.944846 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.945137 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.945141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.945143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.945407 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.945636 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.945794 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.945955 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:38:16.947813 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7623
I20260926 03:38:16.959329 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:16.948498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.1.221.129:41279)
0926 03:38:16.948653 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:16.948699 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:16.948942 (+ 243us) server_negotiation.cc:407] Connection header received
0926 03:38:16.949103 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:16.949107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:16.949161 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:16.949236 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:16.950124 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.950795 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.952143 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:16.952587 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:16.952694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:16.954772 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:16.954778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:16.954781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:16.954810 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:16.957019 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.957587 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.957593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.957596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.957717 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:16.958051 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:16.958055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:16.958059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:16.958305 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:16.958538 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:16.958839 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:38:16.959030 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:38:16.959970 7746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:17.041697 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7623
I20260926 03:38:17.054744 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7757
Sep 26 03:38:17 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:38:17.069993 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:17.055571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.1.221.130:45227)
0926 03:38:17.056137 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:17.056146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:17.056173 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:17.056234 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:17.056241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:17.056457 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:17.056594 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:17.057504 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:17.058651 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:17.059931 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:17.060559 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:17.060687 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:17.064365 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:17.064380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:17.064396 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:17.064439 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:17.067484 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:17.068136 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:17.068144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:17.068147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:17.068284 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:17.068628 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:17.068632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:17.068634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:17.069027 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:17.069261 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:17.069474 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:38:17.069678 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":288,"threads_started":1}
I20260926 03:38:17.071257 7880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:17.150748 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7757
I20260926 03:38:17.163048 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7890
Sep 26 03:38:17 dist-test-slave-lqrk krb5kdc[7517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:38:17.177856 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:17.164066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.1.221.131:33051)
0926 03:38:17.164667 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:17.164674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:17.164702 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:38:17.164783 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:17.164790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:17.165001 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:17.165135 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:17.166116 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:17.167119 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:17.168299 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:17.168859 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:17.168954 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:17.172774 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:17.172785 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:17.172801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:17.172840 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:17.175627 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:17.176217 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:17.176224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:17.176226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:17.176340 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:17.176614 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:17.176617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:17.176619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:17.176949 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:17.177167 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:17.177367 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 03:38:17.177559 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
I20260926 03:38:17.179100 8013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:17.261089 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7890
I20260926 03:38:17.273494 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 7530
I20260926 03:38:17.274550 7592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:17.374830 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 7530
2026-09-26T03:38:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3680 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:17 dist-test-slave-lqrk krb5kdc[8045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:17 dist-test-slave-lqrk krb5kdc[8045](info): set up 2 sockets
Sep 26 03:38:17 dist-test-slave-lqrk krb5kdc[8045](info): commencing operation
krb5kdc: starting...
W20260926 03:38:18.383976 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:18 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
I20260926 03:38:18.584694 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33115
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:18.896613 8058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:18.896916 8058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:18.897032 8058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:18.904243 8058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:18.904348 8058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:18.904399 8058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:18.904476 8058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:18.904533 8058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:18.914318 8058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:18.915613 8058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:18.916867 8058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:18.928314 8064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:18.928349 8066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:18.928447 8063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:18.929417 8058 server_base.cc:1034] running on GCE node
I20260926 03:38:18.930269 8058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:18.931768 8058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:18.932973 8058 hybrid_clock.cc:648] HybridClock initialized: now 1790393898932915 us; error 46 us; skew 500 ppm
I20260926 03:38:18.933302 8058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:18 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:18.936288 8058 init.cc:377] Logged in from keytab as oryx/127.1.221.190@KRBTEST.COM (short username oryx)
I20260926 03:38:18.937976 8058 webserver.cc:460] Webserver started at http://127.1.221.190:35527/ using document root <none> and password file <none>
I20260926 03:38:18.938673 8058 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:18.938802 8058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:18.939086 8058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:18.942070 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "ac94d102a7f647b3a8cc7772a73ff2d8"
format_stamp: "Formatted at 2026-09-26 03:38:18 on dist-test-slave-lqrk"
server_key: "b5e09d3a467a49b58437b6d421755b28"
server_key_iv: "7e81f6aa439a9e4ee021b8d1e9e7c624"
server_key_version: "encryptionkey@0"
I20260926 03:38:18.942859 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "ac94d102a7f647b3a8cc7772a73ff2d8"
format_stamp: "Formatted at 2026-09-26 03:38:18 on dist-test-slave-lqrk"
server_key: "b5e09d3a467a49b58437b6d421755b28"
server_key_iv: "7e81f6aa439a9e4ee021b8d1e9e7c624"
server_key_version: "encryptionkey@0"
I20260926 03:38:18.947753 8058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:38:18.951066 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:18.952481 8058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:18.952653 8058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "ac94d102a7f647b3a8cc7772a73ff2d8"
format_stamp: "Formatted at 2026-09-26 03:38:18 on dist-test-slave-lqrk"
server_key: "b5e09d3a467a49b58437b6d421755b28"
server_key_iv: "7e81f6aa439a9e4ee021b8d1e9e7c624"
server_key_version: "encryptionkey@0"
I20260926 03:38:18.952826 8058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:18.963694 8058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:18.967671 8058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:18.967900 8058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:18.984761 8058 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33115
I20260926 03:38:18.984797 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33115 every 8 connection(s)
I20260926 03:38:18.986100 8058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:18.989797 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:18.993635 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8058
I20260926 03:38:18.993853 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:18.994653 1910 external_mini_cluster.cc:1428] Setting key 9fcab7106c50639fae1d9cfe0b5f7102
I20260926 03:38:19.001376 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Bootstrap starting.
I20260926 03:38:19.005044 8125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:19.006162 8125 log.cc:824] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:19.009615 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: No bootstrap required, opened a new log
Sep 26 03:38:19 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:19.018345 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:18.996694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.1.221.190:33115)
0926 03:38:18.997458 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:18.997471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:18.997510 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:38:18.999792 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:18.999819 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:19.000222 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:19.000556 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:19.002283 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.003663 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.005434 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.006191 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.006362 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:19.011090 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:19.011098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:19.011114 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:19.011155 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:19.014274 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:19.015097 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:19.015104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:19.015111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:19.015277 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:19.015699 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:19.015703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:19.015705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:19.016255 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:19.016496 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:19.016934 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 03:38:19.017322 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":345,"threads_started":1}
I20260926 03:38:19.018182 8125 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } }
I20260926 03:38:19.018568 8125 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:19.018704 8125 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac94d102a7f647b3a8cc7772a73ff2d8, State: Initialized, Role: FOLLOWER
I20260926 03:38:19.019408 8125 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } }
I20260926 03:38:19.019676 8125 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:19.019806 8125 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:19.019980 8125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:19.022317 8125 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } }
I20260926 03:38:19.022763 8125 leader_election.cc:302] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac94d102a7f647b3a8cc7772a73ff2d8; no voters:
I20260926 03:38:19.023289 8125 leader_election.cc:288] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:19.023401 8130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:19.024044 8130 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 LEADER]: Becoming Leader. State: Replica: ac94d102a7f647b3a8cc7772a73ff2d8, State: Running, Role: LEADER
I20260926 03:38:19.024569 8130 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } }
I20260926 03:38:19.025188 8125 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:19.027115 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } }
I20260926 03:38:19.027902 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:19.027603 8132 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94d102a7f647b3a8cc7772a73ff2d8. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } }
I20260926 03:38:19.028272 8132 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:19.032892 8135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:19.038409 8135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:19.047824 8135 catalog_manager.cc:1347] Generated new cluster ID: 1188976a5f3e459a84b373551db5db82
I20260926 03:38:19.047928 8135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:19.065521 8135 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:19.066987 8135 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:19.073385 8135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Generated new TSK 0
I20260926 03:38:19.074173 8135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
I20260926 03:38:19.134940 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:19.454720 8151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:19.455078 8151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:19.455206 8151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:19.462746 8151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:19.462860 8151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:19.463021 8151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:19.472872 8151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:19.474157 8151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:19.475571 8151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:19.488626 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:19.488682 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:19.488652 8156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:19.489890 8151 server_base.cc:1034] running on GCE node
I20260926 03:38:19.490511 8151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:19.491505 8151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:19.492704 8151 hybrid_clock.cc:648] HybridClock initialized: now 1790393899492653 us; error 45 us; skew 500 ppm
I20260926 03:38:19.493022 8151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:19 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393899, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:19.496048 8151 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260926 03:38:19.497476 8151 webserver.cc:460] Webserver started at http://127.1.221.129:35533/ using document root <none> and password file <none>
I20260926 03:38:19.498149 8151 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:19.498255 8151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:19.498545 8151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:19.501449 8151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "3f314af49ade4796a5aeba5fe0caa7d7"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "8a2e6dff63f3809d995424b57edee2a1"
server_key_iv: "06d3e9a7e97362239bc3dba60d881b86"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.502171 8151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "3f314af49ade4796a5aeba5fe0caa7d7"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "8a2e6dff63f3809d995424b57edee2a1"
server_key_iv: "06d3e9a7e97362239bc3dba60d881b86"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.507090 8151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:38:19.510517 8166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:19.511919 8151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:38:19.512051 8151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "3f314af49ade4796a5aeba5fe0caa7d7"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "8a2e6dff63f3809d995424b57edee2a1"
server_key_iv: "06d3e9a7e97362239bc3dba60d881b86"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.512224 8151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:19.529217 8151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:19.533210 8151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:19.533468 8151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:19.534469 8151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:19.535861 8151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:19.535935 8151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:19.536052 8151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:19.536103 8151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:19.562806 8151 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43945
I20260926 03:38:19.562836 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43945 every 8 connection(s)
I20260926 03:38:19.564083 8151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:19.567102 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8151
I20260926 03:38:19.567267 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:19.567896 1910 external_mini_cluster.cc:1428] Setting key a00447d549d9aab7b37e0e9f54f4c88b
Sep 26 03:38:19 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393899, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:19.584110 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:19.566498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59383 (local address 127.1.221.190:33115)
0926 03:38:19.566956 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:19.566963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:19.568046 (+ 1083us) server_negotiation.cc:407] Connection header received
0926 03:38:19.570094 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:19.570099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:19.570182 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:19.570272 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:19.571810 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.572568 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.574150 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.574542 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.574651 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:19.578297 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:19.578303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:19.578307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:19.578342 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:19.580761 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:19.581634 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:19.581637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:19.581640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:19.581764 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:19.582182 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:19.582185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:19.582187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:19.582462 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:19.582571 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:19.583584 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0926 03:38:19.583859 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260926 03:38:19.585446 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:19.567091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:59383)
0926 03:38:19.567890 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:19.567935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:19.569833 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:19.570522 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:19.570531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:19.571027 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:19.571649 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:19.571668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.572707 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.572710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:19.574016 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:19.574028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:19.574755 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:19.574762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:19.574985 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:19.575649 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:19.575675 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:19.578035 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:19.580908 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:19.580918 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:19.580936 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:19.581523 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:19.581852 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:19.581857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:19.581860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:19.582094 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:19.582571 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:19.582579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:19.583049 (+ 470us) client_negotiation.cc:769] Sending connection context
0926 03:38:19.583351 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 03:38:19.583965 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
I20260926 03:38:19.587054 8279 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:19.587345 8279 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:38:19.588100 8279 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
I20260926 03:38:19.590167 8090 ts_manager.cc:194] Registered new tserver with Master: 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:19.591699 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:59383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
I20260926 03:38:19.614925 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:19.939898 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:19.940183 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:19.940317 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:19.947393 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:19.947497 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:19.947643 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:19.957242 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:19.958530 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:19.959822 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:19.971774 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:19.971809 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:19.972400 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:19.972926 8285 server_base.cc:1034] running on GCE node
I20260926 03:38:19.973464 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:19.974310 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:19.975487 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790393899975442 us; error 36 us; skew 500 ppm
I20260926 03:38:19.975807 8285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:19 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393899, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:19.978623 8285 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260926 03:38:19.980046 8285 webserver.cc:460] Webserver started at http://127.1.221.130:40217/ using document root <none> and password file <none>
I20260926 03:38:19.980708 8285 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:19.980826 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:19.981102 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:19.983881 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "c124cb25afc5489d955b6d46ab759ed4"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "a8210bc893521dc0689393707dace24c"
server_key_iv: "d492d2b3b34ab20945935038bf68ca5b"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.984726 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "c124cb25afc5489d955b6d46ab759ed4"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "a8210bc893521dc0689393707dace24c"
server_key_iv: "d492d2b3b34ab20945935038bf68ca5b"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.990887 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:38:19.994971 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:19.996554 8285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:19.996701 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "c124cb25afc5489d955b6d46ab759ed4"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "a8210bc893521dc0689393707dace24c"
server_key_iv: "d492d2b3b34ab20945935038bf68ca5b"
server_key_version: "encryptionkey@0"
I20260926 03:38:19.996928 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:20.012094 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:20.016213 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:20.016481 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:20.017410 8285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:20.018738 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:20.018813 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.018899 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:20.018946 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.048295 8285 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38159
I20260926 03:38:20.048341 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38159 every 8 connection(s)
I20260926 03:38:20.049535 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:20.053695 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8285
I20260926 03:38:20.053865 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:20.054280 1910 external_mini_cluster.cc:1428] Setting key 820b21e2b97837ea42b9b95a5786c866
Sep 26 03:38:20 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393899, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:20.069538 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:20.052001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57479 (local address 127.1.221.190:33115)
0926 03:38:20.052181 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:20.052185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:20.053391 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 03:38:20.055519 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:20.055523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:20.055591 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:20.055686 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:20.057318 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.058040 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.059706 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.060123 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.060237 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:20.063851 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:20.063857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:20.063860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:20.063895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:20.066342 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:20.067076 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:20.067081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:20.067083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:20.067213 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:20.067687 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:20.067690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:20.067692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:20.067971 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:20.068105 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:20.068956 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 03:38:20.069196 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:38:20.070595 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:20.052511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:57479)
0926 03:38:20.053227 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:20.053267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:20.055192 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:20.055976 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:20.055988 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:20.056527 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:20.057111 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:20.057129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.058198 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.058202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:20.059558 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:20.059572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.060350 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.060357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:20.060597 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:20.061249 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:20.061276 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:20.063610 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:20.066492 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:20.066502 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:20.066522 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:20.066947 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:20.067319 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:20.067324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:20.067329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:20.067576 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:20.068094 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:20.068101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:20.068593 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 03:38:20.068847 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:38:20.069441 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":280,"threads_started":1}
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:38:20.072106 8413 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:20.072420 8413 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:20.073144 8413 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
I20260926 03:38:20.074589 8090 ts_manager.cc:194] Registered new tserver with Master: c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
I20260926 03:38:20.075542 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:57479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260926 03:38:20.096887 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:20.406441 8418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:20.406735 8418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:20.406855 8418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:20.414119 8418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:20.414229 8418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:20.414383 8418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:20.425158 8418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:20.426898 8418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:20.428644 8418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:20.444212 8423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:20.444190 8424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:20.445371 8426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:20.445727 8418 server_base.cc:1034] running on GCE node
I20260926 03:38:20.446337 8418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:20.447314 8418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:20.448571 8418 hybrid_clock.cc:648] HybridClock initialized: now 1790393900448513 us; error 52 us; skew 500 ppm
I20260926 03:38:20.448868 8418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:20 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393900, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:20.455175 8418 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260926 03:38:20.456728 8418 webserver.cc:460] Webserver started at http://127.1.221.131:46055/ using document root <none> and password file <none>
I20260926 03:38:20.457453 8418 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:20.457554 8418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:20.457815 8418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:20.460822 8418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "22f159911fb14d44bf7ba4d09ccc6f1e"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "55ed02044d20f68fcaae57d32d934b2d"
server_key_iv: "ea0f684aa24203de97d48917bbc1073c"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.461570 8418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "22f159911fb14d44bf7ba4d09ccc6f1e"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "55ed02044d20f68fcaae57d32d934b2d"
server_key_iv: "ea0f684aa24203de97d48917bbc1073c"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.466553 8418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:20.469821 8433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.471087 8418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:20.471220 8418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "22f159911fb14d44bf7ba4d09ccc6f1e"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "55ed02044d20f68fcaae57d32d934b2d"
server_key_iv: "ea0f684aa24203de97d48917bbc1073c"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.471415 8418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:20.496771 8418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:20.501118 8418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:20.501365 8418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:20.502369 8418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:20.503778 8418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:20.503850 8418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.503938 8418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:20.503980 8418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.532773 8418 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:38907
I20260926 03:38:20.532806 8545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:38907 every 8 connection(s)
I20260926 03:38:20.533978 8418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:20.535231 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8418
I20260926 03:38:20.535423 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:20.535933 1910 external_mini_cluster.cc:1428] Setting key 7fc7282e670adca5e0847df907b96107
Sep 26 03:38:20 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393900, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:20.553385 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:20.536909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:49769 (local address 127.1.221.190:33115)
0926 03:38:20.537097 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:20.537103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:20.538134 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 03:38:20.540179 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:20.540183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:20.540281 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:20.540354 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:20.541834 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.542566 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.544088 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.544489 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.544579 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:20.547797 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:20.547802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:20.547806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:20.547839 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:20.550592 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:20.551373 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:20.551398 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:20.551401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:20.551530 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:20.551947 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:20.551950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:20.551952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:20.552218 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:20.552322 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:20.552982 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 03:38:20.553140 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:38:20.554338 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:20.537184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:49769)
0926 03:38:20.537961 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:20.538011 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:20.539932 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:20.540564 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:20.540573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:20.541115 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:20.541682 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:20.541699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.542671 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.542674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:20.543952 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:20.543964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:20.544658 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:20.544662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:20.544848 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:20.545448 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:20.545468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:20.547604 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:20.550720 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:20.550730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:20.550747 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:20.551258 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:20.551663 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:20.551667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:20.551670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:20.551860 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:20.552314 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:20.552320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:20.552682 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 03:38:20.552917 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:38:20.553304 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
I20260926 03:38:20.555562 8546 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:20.555871 8546 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:20.556588 8546 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
I20260926 03:38:20.557902 8090 ts_manager.cc:194] Registered new tserver with Master: 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:20.558715 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:49769
I20260926 03:38:20.563618 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.189@KRBTEST.COM" created.
I20260926 03:38:20.595295 8279 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.189.keytab.
Entry for principal oryx/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.189.keytab.
I20260926 03:38:20.608863 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:34001
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.189.keytab
--principal=oryx/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33115,127.1.221.189:34001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:20.922436 8551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:20.922720 8551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:20.922818 8551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:20.930116 8551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:20.930235 8551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:20.930280 8551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:20.930335 8551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:20.930377 8551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:20.940320 8551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33115,127.1.221.189:34001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.189.keytab
--principal=oryx/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
W20260926 03:38:20.940653 8551 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.221.190:33115,127.1.221.189:34001'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:38:20.941730 8551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:20.943020 8551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:20.954479 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:20.954517 8559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:20.954550 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:20.955552 8551 server_base.cc:1034] running on GCE node
I20260926 03:38:20.956314 8551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:20.957814 8551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:20.959029 8551 hybrid_clock.cc:648] HybridClock initialized: now 1790393900958957 us; error 43 us; skew 500 ppm
I20260926 03:38:20.959331 8551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:20 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393900, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:20.962365 8551 init.cc:377] Logged in from keytab as oryx/127.1.221.189@KRBTEST.COM (short username oryx)
I20260926 03:38:20.963850 8551 webserver.cc:460] Webserver started at http://127.1.221.189:35117/ using document root <none> and password file <none>
I20260926 03:38:20.964514 8551 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:20.964609 8551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:20.964895 8551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:20.967988 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/instance:
uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "7c98d76b04b084052d2a58658302f90e"
server_key_iv: "1ef5742e8078651a04ec50881602f0fd"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.968745 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal/instance:
uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "7c98d76b04b084052d2a58658302f90e"
server_key_iv: "1ef5742e8078651a04ec50881602f0fd"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.973839 8551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:38:20.977545 8566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:20.978921 8551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:38:20.979094 8551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "7c98d76b04b084052d2a58658302f90e"
server_key_iv: "1ef5742e8078651a04ec50881602f0fd"
server_key_version: "encryptionkey@0"
I20260926 03:38:20.979302 8551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.003151 8551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:21.007086 8551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:21.007326 8551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:21.024305 8551 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:34001
I20260926 03:38:21.024312 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:34001 every 8 connection(s)
I20260926 03:38:21.025954 8551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
I20260926 03:38:21.028276 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8551
I20260926 03:38:21.030108 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:21.040066 8618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393900, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:21.053347 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.030593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.1.221.189:34001)
0926 03:38:21.031424 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.031441 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.031494 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:38:21.033445 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.033476 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.034076 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.034599 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.036158 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.038195 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.039671 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.040765 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.040964 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.045453 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.045462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.045473 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.045514 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.048671 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.049398 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.049407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.049416 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.049642 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.050099 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.050104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.050107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.050835 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.051127 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.051655 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.052097 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260926 03:38:21.057894 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.043154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.1.221.190:33115)
0926 03:38:21.043318 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.043322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.043975 (+ 653us) server_negotiation.cc:407] Connection header received
0926 03:38:21.044151 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.044157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.044233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.044316 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.045514 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.046386 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.047994 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.048635 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.048790 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.052396 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.052402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.052407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.052446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.055238 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.056077 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.056082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.056086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.056283 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.056610 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.056613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.056615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.056948 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.057073 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.057490 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.057668 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:38:21.057926 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.043147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38074)
0926 03:38:21.043835 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.043870 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.044030 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.044411 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.044418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.045096 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.045355 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.045364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.046520 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.046525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.047778 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.047815 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.048843 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.048850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.049024 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.049678 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.049695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.052184 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.055359 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.055367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.055373 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.055939 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.056384 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.056389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.056393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.056527 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.057041 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.057046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.057188 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.057398 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.057631 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"spinlock_wait_cycles":14592,"thread_start_us":322,"threads_started":1}
I20260926 03:38:21.059629 8618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393900, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260926 03:38:21.070698 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.060315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46700)
0926 03:38:21.060466 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.060484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.060582 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.060795 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.060798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.061097 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.061221 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.061229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.062118 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.062122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.062950 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.062960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.063630 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.063635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.063801 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.064247 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.064262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.066072 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.068616 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.068621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.068625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.068957 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.069282 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.069287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.069291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.069410 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.069849 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.069853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.069984 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.070096 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.070360 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:38:21.070847 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.060290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.1.221.189:34001)
0926 03:38:21.060431 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.060435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.060536 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:38:21.060654 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.060657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.060706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.060784 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.061324 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.061995 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.063053 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.063500 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.063585 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.066213 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.066218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.066221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.066247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.068500 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.069053 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.069058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.069062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.069180 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.069499 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.069504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.069507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.069740 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.069867 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.070182 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.070512 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":47}
I20260926 03:38:21.075372 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Bootstrap starting.
I20260926 03:38:21.078045 8413 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
I20260926 03:38:21.078517 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:21.079764 8618 log.cc:824] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:21.082266 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: No bootstrap required, opened a new log
I20260926 03:38:21.089110 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } }
I20260926 03:38:21.089442 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:21.089535 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a2385f1d0d432e85c11f12e4187c3c, State: Initialized, Role: FOLLOWER
I20260926 03:38:21.090101 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } }
I20260926 03:38:21.090883 8624 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } } }
I20260926 03:38:21.091156 8624 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:21.091912 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:38:21.099601 8635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:38:21.099721 8635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:38:21.102677 8551 master_runner.cc:428] Detected that this master a9a2385f1d0d432e85c11f12e4187c3c is joining an existing cluster
I20260926 03:38:21.102764 8551 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.221.189:34001
I20260926 03:38:21.125617 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.106319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.1.221.190:33115)
0926 03:38:21.106481 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.106487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.110866 (+ 4379us) server_negotiation.cc:407] Connection header received
0926 03:38:21.111116 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.111121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.111215 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.111276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.112701 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.113686 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.115583 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.116171 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.116264 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.118722 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.118729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.118734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.118775 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.122183 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.122991 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.122998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.123001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.123142 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.123589 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.123593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.123596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.123935 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.124058 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.124839 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.125052 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:38:21.127061 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.106560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38090)
0926 03:38:21.110763 (+ 4203us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.110781 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.110984 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.111505 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.111509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.112131 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.112521 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.112534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.113871 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.113876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.115425 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.115439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.116551 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.116556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.116808 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.117439 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.117461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.118512 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.122329 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.122334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.122337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.122861 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.123273 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.123277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.123280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.123482 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.124351 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.124356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.124545 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.125882 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.126239 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4059,"mutex_wait_us":64,"thread_start_us":7868,"threads_started":1}
I20260926 03:38:21.132170 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.106582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46706)
0926 03:38:21.114308 (+ 7726us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.114335 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.114765 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.115267 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.115273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.116166 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.116514 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.116536 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.118252 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.118258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.119892 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.119908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.121201 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.121208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.121397 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.121957 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.121989 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.123190 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.129017 (+ 5827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.129022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.129026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.129593 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.130015 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.130020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.130023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.130205 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.130977 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.130985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.131160 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.131303 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.131633 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7569,"spinlock_wait_cycles":21248,"thread_start_us":3867,"threads_started":1}
WARNING: no policy specified for oryx/127.1.221.188@KRBTEST.COM; defaulting to no policy
I20260926 03:38:21.133060 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.107175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.1.221.189:34001)
0926 03:38:21.107361 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.107366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.114417 (+ 7051us) server_negotiation.cc:407] Connection header received
0926 03:38:21.114901 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.114907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.114976 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.115067 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.116679 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.118067 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.120076 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.120792 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.120921 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.123462 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.123469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.123475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.123513 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.128845 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.129735 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.129740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.129744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.129887 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.130332 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.130337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.130340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.130743 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.130915 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.132660 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.132818 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.188@KRBTEST.COM" created.
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260926 03:38:21.153172 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.134513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.1.221.189:34001)
0926 03:38:21.134779 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.134785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.136408 (+ 1623us) server_negotiation.cc:407] Connection header received
0926 03:38:21.136503 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.136509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.136625 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.136727 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.138341 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.139467 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.141427 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.142143 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.142281 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.146273 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.146279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.146283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.146318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.149984 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.150927 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.150932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.150937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.151105 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.151552 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.151556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.151560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.151963 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.152124 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.152705 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.152909 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260926 03:38:21.153666 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.134579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46722)
0926 03:38:21.135962 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.136028 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.136230 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.137028 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.137048 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.137851 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.138159 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.138174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.139652 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.139658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.141243 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.141260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.142306 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.142311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.142520 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.143238 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.143259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.146058 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.150137 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.150152 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.150160 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.150771 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.151224 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.151229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.151233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.151440 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.152117 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.152127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.152350 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.152583 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.152953 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":945,"threads_started":1}
I20260926 03:38:21.155800 8090 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.221.189:34001: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
member_type: NON_VOTER
last_known_addr {
host: "127.1.221.189"
port: 34001
}
attrs {
promote: true
}
}
dest_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8"
cas_config_opid_index: -1
I20260926 03:38:21.156836 8090 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } }
I20260926 03:38:21.159839 8648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } }
I20260926 03:38:21.161280 8593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:21.163132 8648 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.221.189:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.188.keytab.
Entry for principal oryx/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.188.keytab.
I20260926 03:38:21.163872 8551 master.cc:537] Master@127.1.221.189:34001 shutting down...
I20260926 03:38:21.165213 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:45897
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.188.keytab
--principal=oryx/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:38:21.165171 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } }
I20260926 03:38:21.165378 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:21.165977 8593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:38:21.171342 8648 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:38:21.174599 8074 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 03:38:21.176358 8074 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 -> Peer a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189:34001): Couldn't send request to peer a9a2385f1d0d432e85c11f12e4187c3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:38:21.176698 8551 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:21.177155 8551 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:21.177354 8551 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: stopping tablet replica
I20260926 03:38:21.184847 8551 master.cc:552] Master@127.1.221.189:34001 shutdown complete.
I20260926 03:38:21.184952 8551 master_runner.cc:305] Clearing existing system tablet
I20260926 03:38:21.185417 8551 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:21.185523 8551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:21.188762 8551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 03:38:21.190436 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:21.190816 8551 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.191346 8551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 03:38:21.191520 8551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "7c98d76b04b084052d2a58658302f90e"
server_key_iv: "1ef5742e8078651a04ec50881602f0fd"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.196743 8551 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:21.203935 8551 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:38:21.204226 8551 log.cc:1196] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:38:21.204756 8551 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Deleting consensus metadata
I20260926 03:38:21.205055 8551 master_runner.cc:315] Copying system tablet from 127.1.221.190:33115
I20260926 03:38:21.212229 8551 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: tablet copy: Beginning tablet copy session from remote peer at address 127.1.221.190:33115
I20260926 03:38:21.234640 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.213289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.1.221.190:33115)
0926 03:38:21.213477 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.213483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.220034 (+ 6551us) server_negotiation.cc:407] Connection header received
0926 03:38:21.220229 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.220236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.220353 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.220473 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.221524 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.222632 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.224606 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.225271 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.225373 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.228059 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.228066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.228071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.228116 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.231447 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.232272 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.232278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.232281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.232432 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.232915 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.232921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.232925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.233317 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.233476 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.234047 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.234295 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:38:21.234689 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.213160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38092)
0926 03:38:21.219855 (+ 6695us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.219886 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.220073 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.220530 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.220535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.221070 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.221350 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.221362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.222800 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.222806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.224429 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.224446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.225734 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.225740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.225926 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.226614 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.226713 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.227828 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.231596 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.231602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.231607 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.232125 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.232575 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.232580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.232583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.232778 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.233450 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.233454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.233734 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.233909 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.234264 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6534,"thread_start_us":6407,"threads_started":1}
I20260926 03:38:21.235988 8110 tablet_copy_service.cc:140] P ac94d102a7f647b3a8cc7772a73ff2d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a9a2385f1d0d432e85c11f12e4187c3c ({username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:38092)
I20260926 03:38:21.236294 8110 tablet_copy_service.cc:161] P ac94d102a7f647b3a8cc7772a73ff2d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a9a2385f1d0d432e85c11f12e4187c3c at {username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:38092: session id = a9a2385f1d0d432e85c11f12e4187c3c-00000000000000000000000000000000
I20260926 03:38:21.240815 8110 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:38:21.243067 8551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:38:21.250125 8551 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: tablet copy: Starting download of 0 data blocks...
I20260926 03:38:21.250460 8551 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: tablet copy: Starting download of 1 WAL segments...
I20260926 03:38:21.253559 8551 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:38:21.258421 8110 tablet_copy_service.cc:342] P ac94d102a7f647b3a8cc7772a73ff2d8: Request end of tablet copy session a9a2385f1d0d432e85c11f12e4187c3c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:38092
I20260926 03:38:21.258669 8110 tablet_copy_service.cc:433] P ac94d102a7f647b3a8cc7772a73ff2d8: ending tablet copy session a9a2385f1d0d432e85c11f12e4187c3c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a9a2385f1d0d432e85c11f12e4187c3c
I20260926 03:38:21.269232 8551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:21.272074 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:21.272883 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:21.277122 8551 server_base.cc:1034] running on GCE node
W20260926 03:38:21.277186 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:21.277700 8551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:21.278441 8551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:21.279706 8551 hybrid_clock.cc:648] HybridClock initialized: now 1790393901279646 us; error 40 us; skew 500 ppm
I20260926 03:38:21.279932 8551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:21.282037 8551 init.cc:377] Logged in from keytab as oryx/127.1.221.189@KRBTEST.COM (short username oryx)
I20260926 03:38:21.283177 8551 webserver.cc:460] Webserver started at http://127.1.221.189:34901/ using document root <none> and password file <none>
I20260926 03:38:21.283577 8551 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:21.283658 8551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:21.288061 8551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:38:21.289940 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:21.290380 8551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 03:38:21.290550 8551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "7c98d76b04b084052d2a58658302f90e"
server_key_iv: "1ef5742e8078651a04ec50881602f0fd"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.290750 8551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.304615 8551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:21.308508 8551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:21.329056 8551 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:34001
I20260926 03:38:21.329101 8721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:34001 every 8 connection(s)
I20260926 03:38:21.330032 8551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
I20260926 03:38:21.335429 8722 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:38:21.339363 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Bootstrap starting.
I20260926 03:38:21.355400 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:38:21.355937 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: Bootstrap complete.
I20260926 03:38:21.356931 8722 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } }
I20260926 03:38:21.357190 8722 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9a2385f1d0d432e85c11f12e4187c3c, State: Initialized, Role: LEARNER
I20260926 03:38:21.357420 8722 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } }
I20260926 03:38:21.358273 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } }
I20260926 03:38:21.358505 8722 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:21.358690 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: LEARNER
I20260926 03:38:21.372933 8736 catalog_manager.cc:1259] Loaded cluster ID: 1188976a5f3e459a84b373551db5db82
I20260926 03:38:21.373051 8736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: loading cluster ID for follower catalog manager: success
I20260926 03:38:21.376802 8736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: acquiring CA information for follower catalog manager: success
I20260926 03:38:21.379551 8736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:38:21.539703 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:21.539963 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:21.540056 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:21.547327 8650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:21.547482 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:21.547561 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:21.547631 8650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:21.547688 8650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:21.558090 8650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.188.keytab
--principal=oryx/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:21.559588 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:21.560985 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:38:21.561339 8546 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
I20260926 03:38:21.570020 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.563199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46730)
0926 03:38:21.563488 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.563507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.563619 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.564423 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.564427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.564455 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:21.564619 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.564627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.565876 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.565882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.567565 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.567582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.569262 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.569267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.569455 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.569516 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.569601 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.569749 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 03:38:21.570056 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.563560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.1.221.189:34001)
0926 03:38:21.564106 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.564110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.564125 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:38:21.564181 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.564184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.564261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.564360 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:21.564757 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.565744 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.567734 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.569106 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.569223 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.569462 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.569635 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.569765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
I20260926 03:38:21.571049 8697 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 03:38:21.571619 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94d102a7f647b3a8cc7772a73ff2d8. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } }
I20260926 03:38:21.571877 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: LEARNER
W20260926 03:38:21.574196 8742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:21.574155 8743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:21.574717 8745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:21.574980 8650 server_base.cc:1034] running on GCE node
I20260926 03:38:21.575819 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:21.577193 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:21.578363 8650 hybrid_clock.cc:648] HybridClock initialized: now 1790393901578316 us; error 38 us; skew 500 ppm
I20260926 03:38:21.578694 8650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:21.581996 8650 init.cc:377] Logged in from keytab as oryx/127.1.221.188@KRBTEST.COM (short username oryx)
I20260926 03:38:21.583464 8650 webserver.cc:460] Webserver started at http://127.1.221.188:33415/ using document root <none> and password file <none>
I20260926 03:38:21.584169 8650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:21.584286 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:21.584559 8650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:21.587528 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/instance:
uuid: "893121ecd9a9439294f32606274e8c92"
format_stamp: "Formatted at 2026-09-26 03:38:21 on dist-test-slave-lqrk"
server_key: "76f685ac72d903c83144e2598a052a55"
server_key_iv: "8b5fda012f8311ca5e02a1879df3bcd1"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.588274 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal/instance:
uuid: "893121ecd9a9439294f32606274e8c92"
format_stamp: "Formatted at 2026-09-26 03:38:21 on dist-test-slave-lqrk"
server_key: "76f685ac72d903c83144e2598a052a55"
server_key_iv: "8b5fda012f8311ca5e02a1879df3bcd1"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.593497 8650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:38:21.596963 8752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:21.598500 8650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 03:38:21.598662 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "893121ecd9a9439294f32606274e8c92"
format_stamp: "Formatted at 2026-09-26 03:38:21 on dist-test-slave-lqrk"
server_key: "76f685ac72d903c83144e2598a052a55"
server_key_iv: "8b5fda012f8311ca5e02a1879df3bcd1"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.598850 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.613477 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:21.617451 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:21.617684 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:21.635881 8650 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:45897
I20260926 03:38:21.635908 8803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:45897 every 8 connection(s)
I20260926 03:38:21.637281 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
I20260926 03:38:21.639829 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8650
I20260926 03:38:21.642136 8804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:21.651005 8804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:21.664268 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.642022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.1.221.188:45897)
0926 03:38:21.642795 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.642811 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.642864 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:38:21.644505 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.644528 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.644948 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.645302 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.647117 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.648818 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.650502 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.651182 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.651321 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.656043 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.656053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.656066 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.656108 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.659717 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.660543 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.660549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.660556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.660721 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.661119 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.661124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.661127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.661814 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.662107 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.662512 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.662934 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":558,"thread_start_us":344,"threads_started":1}
I20260926 03:38:21.671006 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.654314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.1.221.190:33115)
0926 03:38:21.654515 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.654522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.655042 (+ 520us) server_negotiation.cc:407] Connection header received
0926 03:38:21.655318 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.655324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.655437 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.655555 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.656885 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.657868 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.659901 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.660599 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.660694 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.665626 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.665633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.665638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.665684 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.668571 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.669280 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.669285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.669289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.669437 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.669785 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.669789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.669791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.670106 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.670220 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.670584 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.670761 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:38:21.671018 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.654123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38098)
0926 03:38:21.654918 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.654964 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.655159 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.655578 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.655583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.656378 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.656749 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.656761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.658020 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.658025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.659758 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.659769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.661224 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.661232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.661396 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.662033 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.662054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.665388 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.668686 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.668693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.668698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.669128 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.669544 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.669547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.669549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.669681 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.670199 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.670204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.670349 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.670494 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.670741 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":31,"thread_start_us":410,"threads_started":1}
I20260926 03:38:21.672386 8804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260926 03:38:21.684540 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.673227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46734)
0926 03:38:21.673405 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.673424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.673517 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.673793 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.673797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.674121 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.674253 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.674262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.675217 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.675222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.676296 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.676306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.676961 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.676966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.677126 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.677608 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.677623 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.679504 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.682236 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.682240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.682242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.682581 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.682865 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.682868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.682870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.682978 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.683874 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.683877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.683994 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.684099 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.684315 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:38:21.684707 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.673313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.1.221.189:34001)
0926 03:38:21.673489 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.673494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.673512 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:21.673624 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.673628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.673689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.673765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.674423 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.675098 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.676424 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.676828 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.676933 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.679676 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.679681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.679684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.679719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.682114 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.682669 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.682672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.682674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.682790 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.683049 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.683052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.683054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.683772 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.683878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.684244 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.684422 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:38:21.685647 8804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260926 03:38:21.696890 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.686203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45024)
0926 03:38:21.686340 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.686354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.686429 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.686736 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.686739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.687064 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.687206 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.687214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.688183 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.688186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.689009 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.689019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.689658 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.689662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.689810 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.690317 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.690333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.692325 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.694802 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.694806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.694809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.695151 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.695495 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.695499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.695501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.695686 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.696117 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.696120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.696239 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.696356 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.696586 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:38:21.696972 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.686361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.1.221.188:45897)
0926 03:38:21.686501 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.686505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.686519 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:38:21.686564 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.686568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.686618 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.686688 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.687322 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.688069 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.689142 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.689527 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.689624 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.692491 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.692495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.692498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.692526 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.694663 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.695246 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.695249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.695251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.695362 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.695783 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.695786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.695788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.696031 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.696129 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.696425 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.696715 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":50}
I20260926 03:38:21.701689 8804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Bootstrap starting.
I20260926 03:38:21.704885 8804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:21.706274 8804 log.cc:824] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:21.708890 8804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: No bootstrap required, opened a new log
I20260926 03:38:21.715790 8804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } }
I20260926 03:38:21.716100 8804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:21.716190 8804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893121ecd9a9439294f32606274e8c92, State: Initialized, Role: FOLLOWER
I20260926 03:38:21.716706 8804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } }
I20260926 03:38:21.717460 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } } }
I20260926 03:38:21.717760 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:21.718699 8804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:38:21.727025 8821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:38:21.727156 8821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:38:21.732618 8650 master_runner.cc:428] Detected that this master 893121ecd9a9439294f32606274e8c92 is joining an existing cluster
I20260926 03:38:21.732767 8650 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.221.188:45897
I20260926 03:38:21.754487 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.737306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.1.221.189:34001)
0926 03:38:21.737481 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.737487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.738064 (+ 577us) server_negotiation.cc:407] Connection header received
0926 03:38:21.738937 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.738942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.739011 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.739072 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.740843 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.741881 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.743899 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.744526 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.744624 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.747691 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.747697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.747702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.747754 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.750901 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.751651 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.751656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.751660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.751807 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.752238 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.752243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.752246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.752585 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.752711 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.753395 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.754154 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:38:21.755049 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.738423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.1.221.188:45897)
0926 03:38:21.738541 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.738547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.740122 (+ 1575us) server_negotiation.cc:407] Connection header received
0926 03:38:21.740265 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.740270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.740337 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.740427 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.741677 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.742959 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.744857 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.745990 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.746087 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.748417 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.748424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.748429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.748463 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.752088 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.752761 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.752765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.752768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.752927 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.753358 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.753364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.753366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.753821 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.753984 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.754474 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.754683 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":33}
I20260926 03:38:21.755733 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.737186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46746)
0926 03:38:21.737961 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.737982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.738805 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.739251 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.739255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.740288 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.740703 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.740716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.741995 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.742001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.743718 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.743740 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.744923 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.744929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.745112 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.746249 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.746271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.747471 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.751042 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.751047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.751051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.751521 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.751941 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.751945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.751952 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.752116 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.752774 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.752778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.752995 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.754688 (+ 1693us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.754992 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":469,"threads_started":1}
I20260926 03:38:21.755795 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.738125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45030)
0926 03:38:21.740031 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.740048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.740163 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.740596 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.740601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.741224 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.741532 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.741547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.743134 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.743139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.744717 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.744729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.746396 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.746402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.746555 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.747139 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.747161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.748239 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.752207 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.752211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.752214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.752643 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.753049 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.753053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.753056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.753178 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.754072 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.754076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.754227 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.755080 (+ 853us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.755324 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":1666,"threads_started":1}
I20260926 03:38:21.758361 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.737213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38104)
0926 03:38:21.738509 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.738532 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.738659 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.739129 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.739133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.741648 (+ 2515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.742499 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.742513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.744180 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.744185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.746186 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.746203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.747415 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.747421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.747598 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.748212 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.748234 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.749542 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.753195 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.753200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.753204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.753699 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.754139 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.754143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.754147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.754319 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.755680 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.755688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.757507 (+ 1819us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.757663 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.757940 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":27,"thread_start_us":993,"threads_started":1}
I20260926 03:38:21.758450 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.736975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.1.221.190:33115)
0926 03:38:21.737151 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.737157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.738600 (+ 1443us) server_negotiation.cc:407] Connection header received
0926 03:38:21.738788 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.738794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.738874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.738956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.742716 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.743993 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.746375 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.747004 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.747106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.749780 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.749786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.749791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.749833 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.753049 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.753848 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.753854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.753857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.754002 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.754449 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.754454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.754458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.754783 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.754895 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.758009 (+ 3114us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.758189 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:38:21.770467 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8151
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260926 03:38:21.778685 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.763466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45034)
0926 03:38:21.763641 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.763660 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.763769 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.764075 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.764080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.764500 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.764730 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.764743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.765692 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.765698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.767575 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.767589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.768504 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.768511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.768692 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.769263 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.769286 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.771645 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.775781 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.775786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.775790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.776280 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.776609 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.776614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.776618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.777150 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.777690 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.777696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.777875 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.778023 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.778307 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:38:21.778693 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.763649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.1.221.188:45897)
0926 03:38:21.763789 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.763793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.763807 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:38:21.763881 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.763884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.763958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.764027 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.764860 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.765559 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.767731 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.768344 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.768462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.771875 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.771882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.771888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.771934 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.775640 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.776392 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.776398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.776402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.776508 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.777265 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.777269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.777271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.777558 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.777699 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.778134 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.778361 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:38:21.780316 8090 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.221.188:45897: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "893121ecd9a9439294f32606274e8c92"
member_type: NON_VOTER
last_known_addr {
host: "127.1.221.188"
port: 45897
}
attrs {
promote: true
}
}
dest_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8"
cas_config_opid_index: 5
I20260926 03:38:21.781376 8090 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } }
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.129@KRBTEST.COM
I20260926 03:38:21.782960 8648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 893121ecd9a9439294f32606274e8c92 (127.1.221.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.784238 8779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:21.787765 8779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:38:21.792444 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.771481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.1.221.129:43945)
0926 03:38:21.773073 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.773080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.773104 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:38:21.773161 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.773166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.773369 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.773528 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.774803 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.776012 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.777557 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.778311 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.779403 (+ 1092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.784223 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.784238 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.784256 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.784298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.788204 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.789042 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.789051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.789054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.789199 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.789660 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.789677 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.789680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.790101 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.791541 (+ 1440us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.791781 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.792041 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":1278,"threads_started":1}
I20260926 03:38:21.794145 8274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:21.803277 8648 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.221.188:45897
I20260926 03:38:21.804033 8650 master.cc:537] Master@127.1.221.188:45897 shutting down...
I20260926 03:38:21.809721 8833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:21.817163 8697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:38:21.818739 8833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:38:21.834077 8831 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.834499 8831 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:21.835733 8697 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 893121ecd9a9439294f32606274e8c92 (127.1.221.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.840583 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: true } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.840883 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: LEARNER
I20260926 03:38:21.841825 8648 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: attempting to promote NON_VOTER a9a2385f1d0d432e85c11f12e4187c3c to VOTER
I20260926 03:38:21.842490 8648 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } }
I20260926 03:38:21.847630 8650 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:21.848088 8650 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:21.848318 8650 tablet_replica.cc:330] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: stopping tablet replica
I20260926 03:38:21.850497 8697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 LEARNER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:38:21.855849 8650 master.cc:552] Master@127.1.221.188:45897 shutdown complete.
I20260926 03:38:21.856006 8650 master_runner.cc:305] Clearing existing system tablet
I20260926 03:38:21.856422 8650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:21.856591 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:21.857740 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.857312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45042)
0926 03:38:21.857462 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.857573 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 03:38:21.858026 8074 consensus_peers.cc:487] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 -> Peer 893121ecd9a9439294f32606274e8c92 (127.1.221.188:45897): Couldn't send request to peer 893121ecd9a9439294f32606274e8c92. Status: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:38:21.859218 8831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 03:38:21.860039 8650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:38:21.862186 8835 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.862794 8697 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.866428 8833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.866710 8833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:21.867085 8833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.867120 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:21.867305 8833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:21.867348 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:21.871745 8841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:21.872107 8650 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.872380 8650 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.000s
I20260926 03:38:21.872479 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "893121ecd9a9439294f32606274e8c92"
format_stamp: "Formatted at 2026-09-26 03:38:21 on dist-test-slave-lqrk"
server_key: "76f685ac72d903c83144e2598a052a55"
server_key_iv: "8b5fda012f8311ca5e02a1879df3bcd1"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.876055 8650 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:21.880518 8650 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:38:21.880648 8650 log.cc:1196] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:38:21.881038 8650 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Deleting consensus metadata
I20260926 03:38:21.881234 8650 master_runner.cc:315] Copying system tablet from 127.1.221.190:33115
I20260926 03:38:21.883566 8650 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: tablet copy: Beginning tablet copy session from remote peer at address 127.1.221.190:33115
I20260926 03:38:21.898264 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.884307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.1.221.190:33115)
0926 03:38:21.884561 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:21.884567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:21.887914 (+ 3347us) server_negotiation.cc:407] Connection header received
0926 03:38:21.888089 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:21.888095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:21.888166 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:21.888256 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:21.889048 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.889814 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.891040 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.891505 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.891611 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.893458 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:21.893463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:21.893466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:21.893496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:21.895914 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.896473 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.896478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.896481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.896603 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:21.896942 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:21.896946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:21.896950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:21.897200 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:21.897343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:21.897704 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 03:38:21.897900 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:38:21.898331 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:21.884282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38112)
0926 03:38:21.887786 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:21.887807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:21.887964 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:21.888299 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:21.888303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:21.888696 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:21.888902 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.888911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.889946 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.889951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.890886 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:21.890898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:21.891649 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:21.891654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:21.891841 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:21.892485 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:21.892534 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:21.893272 (+ 738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:21.896032 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.896037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.896042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.896370 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.896697 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:21.896702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:21.896706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:21.896845 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:21.897314 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:21.897322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:21.897473 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 03:38:21.897592 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:38:21.897900 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3373,"thread_start_us":3238,"threads_started":1}
I20260926 03:38:21.899087 8110 tablet_copy_service.cc:140] P ac94d102a7f647b3a8cc7772a73ff2d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 893121ecd9a9439294f32606274e8c92 ({username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:38112)
I20260926 03:38:21.899245 8110 tablet_copy_service.cc:161] P ac94d102a7f647b3a8cc7772a73ff2d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 893121ecd9a9439294f32606274e8c92 at {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:38112: session id = 893121ecd9a9439294f32606274e8c92-00000000000000000000000000000000
I20260926 03:38:21.902030 8110 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:38:21.903581 8650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:38:21.908129 8650 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: tablet copy: Starting download of 0 data blocks...
I20260926 03:38:21.908378 8650 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: tablet copy: Starting download of 1 WAL segments...
I20260926 03:38:21.910472 8650 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:38:21.913690 8110 tablet_copy_service.cc:342] P ac94d102a7f647b3a8cc7772a73ff2d8: Request end of tablet copy session 893121ecd9a9439294f32606274e8c92-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:38112
I20260926 03:38:21.913817 8110 tablet_copy_service.cc:433] P ac94d102a7f647b3a8cc7772a73ff2d8: ending tablet copy session 893121ecd9a9439294f32606274e8c92-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 893121ecd9a9439294f32606274e8c92
I20260926 03:38:21.923045 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:21.925593 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:21.930395 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:21.936560 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8151
I20260926 03:38:21.940043 8650 server_base.cc:1034] running on GCE node
W20260926 03:38:21.943656 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:21.944116 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:21.944782 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:21.945981 8650 hybrid_clock.cc:648] HybridClock initialized: now 1790393901945931 us; error 44 us; skew 500 ppm
I20260926 03:38:21.946218 8650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:21 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393901, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:21.948361 8650 init.cc:377] Logged in from keytab as oryx/127.1.221.188@KRBTEST.COM (short username oryx)
I20260926 03:38:21.951897 8650 webserver.cc:460] Webserver started at http://127.1.221.188:46745/ using document root <none> and password file <none>
I20260926 03:38:21.952225 8650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:21.952322 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:21.954910 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:43945
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:33115
--webserver_port=35533
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:38:21.955047 8650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:38:21.956775 8858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:21.957190 8650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:38:21.957304 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "893121ecd9a9439294f32606274e8c92"
format_stamp: "Formatted at 2026-09-26 03:38:21 on dist-test-slave-lqrk"
server_key: "76f685ac72d903c83144e2598a052a55"
server_key_iv: "8b5fda012f8311ca5e02a1879df3bcd1"
server_key_version: "encryptionkey@0"
I20260926 03:38:21.957473 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:21.970105 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:21.974068 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:22.041415 8650 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:45897
I20260926 03:38:22.041452 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:45897 every 8 connection(s)
I20260926 03:38:22.042605 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
I20260926 03:38:22.048123 8911 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:38:22.051780 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Bootstrap starting.
I20260926 03:38:22.075305 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:38:22.076085 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Bootstrap complete.
I20260926 03:38:22.077170 8911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } }
I20260926 03:38:22.077499 8911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 893121ecd9a9439294f32606274e8c92, State: Initialized, Role: LEARNER
I20260926 03:38:22.077805 8911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } }
I20260926 03:38:22.079835 8914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:22.080260 8914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:38:22.080166 8911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:22.104460 8925 catalog_manager.cc:1259] Loaded cluster ID: 1188976a5f3e459a84b373551db5db82
I20260926 03:38:22.104604 8925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: loading cluster ID for follower catalog manager: success
I20260926 03:38:22.108196 8925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: acquiring CA information for follower catalog manager: success
I20260926 03:38:22.110682 8925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:38:22.296618 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.289821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45050)
0926 03:38:22.290020 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:22.290077 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:22.290242 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:22.290823 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:22.290829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:22.290860 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:22.291067 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.291079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.292569 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.292575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.294480 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.294498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.295803 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.295810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.296014 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.296062 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:38:22.296166 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:38:22.296313 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:38:22.296742 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.289901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.1.221.188:45897)
0926 03:38:22.290490 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:22.290494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:22.290512 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:22.290573 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:22.290577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:22.290648 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:22.290775 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:22.291217 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.292417 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.294678 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.295596 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.295722 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.295926 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:22.296210 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 03:38:22.296324 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260926 03:38:22.297636 8886 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 03:38:22.298193 8914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94d102a7f647b3a8cc7772a73ff2d8. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:22.298470 8914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: LEARNER
W20260926 03:38:22.337181 8863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:22.337592 8863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:22.337823 8863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:22.349211 8863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:22.349318 8863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:22.349436 8863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:22.359731 8863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:43945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=35533
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:22.361029 8863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:22.362329 8863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:22.375370 8934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:22.375429 8931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:22.375485 8932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:22.380630 8863 server_base.cc:1034] running on GCE node
I20260926 03:38:22.381176 8863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:22.382074 8863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:22.383265 8863 hybrid_clock.cc:648] HybridClock initialized: now 1790393902383221 us; error 40 us; skew 500 ppm
I20260926 03:38:22.383960 8863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:22.386852 8863 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260926 03:38:22.388353 8863 webserver.cc:460] Webserver started at http://127.1.221.129:35533/ using document root <none> and password file <none>
I20260926 03:38:22.389098 8863 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:22.389216 8863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:22.394527 8863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:38:22.397173 8941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:22.398483 8863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:38:22.398638 8863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "3f314af49ade4796a5aeba5fe0caa7d7"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "8a2e6dff63f3809d995424b57edee2a1"
server_key_iv: "06d3e9a7e97362239bc3dba60d881b86"
server_key_version: "encryptionkey@0"
I20260926 03:38:22.399681 8863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:22.415235 8863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:22.418964 8863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:22.419203 8863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:22.420249 8863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:22.421569 8863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:22.421656 8863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:22.421770 8863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:22.421844 8863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:22.456097 8863 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43945
I20260926 03:38:22.456166 9053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43945 every 8 connection(s)
I20260926 03:38:22.457515 8863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:22.460994 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 8863
I20260926 03:38:22.461372 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8285
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.130@KRBTEST.COM
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:22.493801 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.462845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.1.221.130:38159)
0926 03:38:22.468276 (+ 5431us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:22.468290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:22.472245 (+ 3955us) server_negotiation.cc:407] Connection header received
0926 03:38:22.472381 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:22.472390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:22.472654 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:22.472811 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:22.474505 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.475780 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.477318 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.478155 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.478259 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.484215 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:22.484230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:22.484247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:22.484290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:22.488409 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.489205 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.489227 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.489231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.489379 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.491416 (+ 2037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.491421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.491425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.491855 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:22.492238 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:22.492473 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:38:22.492728 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5179,"thread_start_us":5051,"threads_started":1}
I20260926 03:38:22.495273 8408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:22.497274 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.476074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:37357 (local address 127.1.221.188:45897)
0926 03:38:22.476257 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:22.476262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:22.477423 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 03:38:22.477623 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:22.477628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:22.477699 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:22.477762 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:22.479892 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.480951 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484174 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.484800 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.490259 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:22.490265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:22.490270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:22.490308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:22.493261 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.494472 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.494478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.494481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.494625 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.495170 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.495176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.495180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.495567 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:22.495707 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:22.496724 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0926 03:38:22.496944 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:38:22.497974 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.460910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:57107 (local address 127.1.221.189:34001)
0926 03:38:22.478910 (+ 18000us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:22.478915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:22.478934 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:22.478979 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:22.478983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:22.479074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:22.479179 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:22.480166 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.481200 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484199 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.484800 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.489755 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:22.489762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:22.489767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:22.489808 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:22.495297 (+ 5489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.495905 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.495910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.495914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.496029 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.496390 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.496395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.496399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.496686 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:22.496823 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:22.497578 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 03:38:22.497736 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17892,"thread_start_us":17770,"threads_started":1}
I20260926 03:38:22.498704 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.476104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.1.221.129:37357)
0926 03:38:22.477341 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:22.477357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:22.477506 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:22.477955 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:22.477960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:22.478787 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:22.479675 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.479706 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.481150 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.481154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.483670 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.484032 (+ 362us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.485243 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.485254 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.485641 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.486371 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:22.486418 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:22.490020 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:22.493395 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.493404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.493423 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.494335 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.494769 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.494774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.494778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.495042 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.495808 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:22.495817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:22.496283 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 03:38:22.496609 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 03:38:22.497179 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":313,"spinlock_wait_cycles":8320,"thread_start_us":2332,"threads_started":1}
I20260926 03:38:22.498723 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.461561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.1.221.129:57107)
0926 03:38:22.468941 (+ 7380us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:22.468996 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:22.472751 (+ 3755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:22.479218 (+ 6467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:22.479223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:22.479696 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:22.479994 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.480006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.481360 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.481364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.484066 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.484077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.485246 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.485254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.485670 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.486371 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:22.486417 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:22.489490 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:22.495468 (+ 5978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.495471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.495473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.495786 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.496126 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.496131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.496134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.496281 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.496881 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:22.496886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:22.497067 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 03:38:22.498138 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0926 03:38:22.498372 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7018,"mutex_wait_us":329,"spinlock_wait_cycles":7552,"thread_start_us":13822,"threads_started":1}
I20260926 03:38:22.500275 9055 heartbeater.cc:347] Connected to a master server at 127.1.221.188:45897
I20260926 03:38:22.500347 9056 heartbeater.cc:347] Connected to a master server at 127.1.221.189:34001
I20260926 03:38:22.500614 9056 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:22.500620 9055 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:22.501617 9056 heartbeater.cc:511] Master 127.1.221.189:34001 requested a full tablet report, sending...
I20260926 03:38:22.501644 9055 heartbeater.cc:511] Master 127.1.221.188:45897 requested a full tablet report, sending...
I20260926 03:38:22.503108 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.462272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:58161)
0926 03:38:22.474969 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:22.474994 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:22.475131 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:22.477701 (+ 2570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:22.477715 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:22.479688 (+ 1973us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:22.480094 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.480107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.481715 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.481720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.484066 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:22.484080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.485275 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.485283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:22.485657 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.486508 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:22.486531 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:22.490288 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:22.493707 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.493712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.493716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.494373 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.494769 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:22.494774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:22.494778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:22.501175 (+ 6397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:22.502017 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:22.502021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:22.502193 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:38:22.502406 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 03:38:22.502661 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12566,"mutex_wait_us":192,"spinlock_wait_cycles":26752,"thread_start_us":6200,"threads_started":1}
I20260926 03:38:22.503211 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:22.463173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58161 (local address 127.1.221.190:33115)
0926 03:38:22.467753 (+ 4580us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:22.467760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:22.475091 (+ 7331us) server_negotiation.cc:407] Connection header received
0926 03:38:22.476949 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:22.476956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:22.477065 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:22.477150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:22.480306 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.481485 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484224 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:22.484841 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:22.484938 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:22.490469 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:22.490474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:22.490477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:22.490509 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:22.493583 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.494489 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.494494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.494497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.494642 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:22.501325 (+ 6683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:22.501331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:22.501334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:22.501759 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:22.501966 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:22.502554 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 03:38:22.502790 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4434,"thread_start_us":4329,"threads_started":1}
I20260926 03:38:22.504009 8687 ts_manager.cc:194] Registered new tserver with Master: 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:22.504241 8876 ts_manager.cc:194] Registered new tserver with Master: 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:22.504537 9054 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:22.504657 9054 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:22.505120 9054 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
I20260926 03:38:22.506119 8090 ts_manager.cc:194] Re-registered known tserver with Master: 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:22.506888 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:58161
I20260926 03:38:22.590528 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8285
I20260926 03:38:22.603343 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:38159
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:33115
--webserver_port=40217
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:38:22.884474 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: true } } }
I20260926 03:38:22.884855 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:22.886363 9067 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: attempting to promote NON_VOTER 893121ecd9a9439294f32606274e8c92 to VOTER
I20260926 03:38:22.886933 9067 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } }
I20260926 03:38:22.889202 8697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:38:22.889206 8886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 LEARNER]: Refusing update from remote peer ac94d102a7f647b3a8cc7772a73ff2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:38:22.889801 8835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:38:22.890237 9067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:38:22.892930 9067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 893121ecd9a9439294f32606274e8c92 (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.895033 8886 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 893121ecd9a9439294f32606274e8c92 (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.895918 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.896095 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:22.896303 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.896466 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:22.896889 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.897075 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:22.897353 8697 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 893121ecd9a9439294f32606274e8c92 (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.900202 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.900545 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:22.903249 9073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:22.903620 9073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:38:22.923628 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:22.923918 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:22.924048 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:22.931128 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:22.931239 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:22.931420 9066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:22.941043 9066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:38159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=40217
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:22.942306 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:22.943612 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:22.956277 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:22.956346 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:22.956329 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:22.957166 9066 server_base.cc:1034] running on GCE node
I20260926 03:38:22.957803 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:22.958830 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:22.960039 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790393902959965 us; error 70 us; skew 500 ppm
I20260926 03:38:22.960355 9066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:22 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:22.963320 9066 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260926 03:38:22.964737 9066 webserver.cc:460] Webserver started at http://127.1.221.130:40217/ using document root <none> and password file <none>
I20260926 03:38:22.965413 9066 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:22.965534 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:22.970736 9066 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:38:22.973459 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:22.974833 9066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:22.974983 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "c124cb25afc5489d955b6d46ab759ed4"
format_stamp: "Formatted at 2026-09-26 03:38:19 on dist-test-slave-lqrk"
server_key: "a8210bc893521dc0689393707dace24c"
server_key_iv: "d492d2b3b34ab20945935038bf68ca5b"
server_key_version: "encryptionkey@0"
I20260926 03:38:22.976022 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:23.004706 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:23.008879 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:23.009147 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:23.010129 9066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:23.011515 9066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:23.011610 9066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:23.011709 9066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:23.011775 9066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:23.041481 9066 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38159
I20260926 03:38:23.041538 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38159 every 8 connection(s)
I20260926 03:38:23.043416 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:23.052287 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9066
I20260926 03:38:23.052597 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8418
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393902, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.131@KRBTEST.COM
I20260926 03:38:23.087141 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.052234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:42673 (local address 127.1.221.189:34001)
0926 03:38:23.052939 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.052945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.063885 (+ 10940us) server_negotiation.cc:407] Connection header received
0926 03:38:23.064387 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.064394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.064483 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.064566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.067447 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.069577 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.072277 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.072900 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.072997 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.078817 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.078824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.078829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.078882 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.082250 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084204 (+ 1954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.084209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.084213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.084361 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084885 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.084890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.084893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.085258 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.085410 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.086537 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.086775 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260926 03:38:23.088052 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.046840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59595 (local address 127.1.221.188:45897)
0926 03:38:23.067266 (+ 20426us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.067271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.067289 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:23.067429 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.067434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.067524 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.068844 (+ 1320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.069962 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.071047 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.073204 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.073801 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.074017 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.078823 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.078830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.078834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.078875 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.084807 (+ 5932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.085503 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.085508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.085511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.085663 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.086079 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.086084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.086087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.086391 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.086501 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.087484 (+ 983us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.087701 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20292,"thread_start_us":20143,"threads_started":1}
I20260926 03:38:23.088460 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.047214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:59053)
0926 03:38:23.059196 (+ 11982us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.059224 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.059878 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.069902 (+ 10024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.069908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.070620 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.071613 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.071634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.073235 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.073240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.074825 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.074840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.075887 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.075895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.076111 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.076739 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.076763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.079274 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.083217 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.083227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.083247 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.084111 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.084584 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.084593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.084786 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.085498 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.085503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.086076 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.086414 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.086952 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4543,"mutex_wait_us":80,"spinlock_wait_cycles":27392,"thread_start_us":681,"threads_started":1}
I20260926 03:38:23.088496 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.052819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.1.221.130:42673)
0926 03:38:23.063748 (+ 10929us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.063775 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.064221 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.065167 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.065182 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.066233 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.067174 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.067203 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.069802 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.069807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.072110 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.072124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.073358 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.073365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.073701 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.075708 (+ 2007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.075730 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.078534 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.083276 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.083282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.083286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.084041 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.084503 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.084512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.084757 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.085493 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.085501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.086076 (+ 575us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.087335 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.087614 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10763,"mutex_wait_us":85,"spinlock_wait_cycles":40192,"thread_start_us":11101,"threads_started":1}
I20260926 03:38:23.088620 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.047214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.1.221.130:59595)
0926 03:38:23.048478 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.048530 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.051115 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.067697 (+ 16582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.067703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.068150 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.068590 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.068604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.071566 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.071572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.073008 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.073020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.074049 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.074057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.074228 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.074952 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.074982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.078534 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.084945 (+ 6411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.084952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.085377 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.085789 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.085793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.085796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.085956 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.086564 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.086568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.086741 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.086864 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.087130 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":1,"thread_start_us":4315,"threads_started":1}
I20260926 03:38:23.089521 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.046486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59053 (local address 127.1.221.190:33115)
0926 03:38:23.069547 (+ 23061us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.069553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.069570 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:38:23.069626 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.069630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.069706 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.069815 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.071855 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.072998 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.075044 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.075641 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.076479 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.079515 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.079522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.079526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.079577 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.083056 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084287 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.084293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.084297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.084452 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.084917 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.084922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.084926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.085288 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.085420 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.088889 (+ 3469us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.089108 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22939,"thread_start_us":22819,"threads_started":1}
I20260926 03:38:23.090036 9204 heartbeater.cc:347] Connected to a master server at 127.1.221.189:34001
I20260926 03:38:23.090147 9203 heartbeater.cc:347] Connected to a master server at 127.1.221.188:45897
I20260926 03:38:23.090263 9202 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:23.090376 9204 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.090404 9202 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.090380 9203 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.091418 9203 heartbeater.cc:511] Master 127.1.221.188:45897 requested a full tablet report, sending...
I20260926 03:38:23.092118 9202 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
I20260926 03:38:23.092404 9204 heartbeater.cc:511] Master 127.1.221.189:34001 requested a full tablet report, sending...
I20260926 03:38:23.093119 8090 ts_manager.cc:194] Re-registered known tserver with Master: c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159)
I20260926 03:38:23.093146 8876 ts_manager.cc:194] Registered new tserver with Master: c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159)
I20260926 03:38:23.093436 8687 ts_manager.cc:194] Registered new tserver with Master: c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159)
I20260926 03:38:23.094204 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:59053
I20260926 03:38:23.096771 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.053579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.1.221.131:38907)
0926 03:38:23.074694 (+ 21115us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.074703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.074734 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:38:23.074801 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.074811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.075147 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.075319 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.077958 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.079633 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.081695 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.082286 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.082401 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.088515 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.089070 (+ 555us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.089093 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.089138 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.093708 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.094585 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.094594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.094598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.094770 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.095175 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.095180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.095183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.095608 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.095905 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.096133 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.096386 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20875,"thread_start_us":20719,"threads_started":1}
I20260926 03:38:23.097972 8541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:23.179854 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8418
I20260926 03:38:23.193677 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:38907
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:33115
--webserver_port=46055
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:38:23.509694 9054 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
W20260926 03:38:23.514452 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:23.514753 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:23.514890 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:23.522185 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:23.522317 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:23.522465 9215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:23.532197 9215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:38907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=46055
--tserver_master_addrs=127.1.221.190:33115,127.1.221.189:34001,127.1.221.188:45897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:23.533468 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:23.534751 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:23.547333 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:23.547562 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:23.547850 9223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:23.548058 9215 server_base.cc:1034] running on GCE node
I20260926 03:38:23.548653 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:23.549566 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:23.550765 9215 hybrid_clock.cc:648] HybridClock initialized: now 1790393903550708 us; error 48 us; skew 500 ppm
I20260926 03:38:23.551071 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393903, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:23.554034 9215 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260926 03:38:23.555428 9215 webserver.cc:460] Webserver started at http://127.1.221.131:46055/ using document root <none> and password file <none>
I20260926 03:38:23.556079 9215 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:23.556193 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:23.561606 9215 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 03:38:23.564265 9230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:23.565526 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:23.565665 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "22f159911fb14d44bf7ba4d09ccc6f1e"
format_stamp: "Formatted at 2026-09-26 03:38:20 on dist-test-slave-lqrk"
server_key: "55ed02044d20f68fcaae57d32d934b2d"
server_key_iv: "ea0f684aa24203de97d48917bbc1073c"
server_key_version: "encryptionkey@0"
I20260926 03:38:23.566589 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:23.596251 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:23.600272 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:23.600517 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:23.601473 9215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:23.602797 9215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:23.602880 9215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:23.602977 9215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:23.603047 9215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:23.632899 9215 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:38907
I20260926 03:38:23.632977 9342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:38907 every 8 connection(s)
I20260926 03:38:23.634223 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:23.642910 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9215
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393903, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393903, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:23.674305 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.637012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:56913 (local address 127.1.221.189:34001)
0926 03:38:23.643604 (+ 6592us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.643611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.648472 (+ 4861us) server_negotiation.cc:407] Connection header received
0926 03:38:23.650969 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.650976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.651069 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.651173 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.653826 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.654598 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.657262 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.657790 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.657907 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.665817 (+ 7910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.665825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.665831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.665891 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.669877 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.670860 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.670866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.670870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.671055 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.671748 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.671754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.671757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.672148 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.672291 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.673498 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.673843 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6453,"thread_start_us":6349,"threads_started":1}
I20260926 03:38:23.675401 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.637774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.1.221.131:56913)
0926 03:38:23.648362 (+ 10588us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.648380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.650647 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.651843 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.651849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.652763 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.653625 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.653651 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.654748 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.654753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.657131 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.657146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.658038 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.658048 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.658378 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.659407 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.659443 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.665521 (+ 6078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.670016 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.670027 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.670048 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.670750 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.671352 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.671357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.671361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.671644 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.672609 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.672630 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.673084 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.673382 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.673950 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10444,"mutex_wait_us":34,"spinlock_wait_cycles":43776,"thread_start_us":10560,"threads_started":1}
I20260926 03:38:23.676916 9345 heartbeater.cc:347] Connected to a master server at 127.1.221.189:34001
I20260926 03:38:23.677309 9345 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.678268 9345 heartbeater.cc:511] Master 127.1.221.189:34001 requested a full tablet report, sending...
Sep 26 03:38:23 dist-test-slave-lqrk krb5kdc[8045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393903, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260926 03:38:23.680284 8687 ts_manager.cc:194] Registered new tserver with Master: 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:23.685920 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.637241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57953 (local address 127.1.221.190:33115)
0926 03:38:23.647927 (+ 10686us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.647933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.651070 (+ 3137us) server_negotiation.cc:407] Connection header received
0926 03:38:23.651459 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.651465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.651570 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.651681 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.654712 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.655948 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.662514 (+ 6566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.663214 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.663335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.677739 (+ 14404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.677748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.677754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.677806 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.682252 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.683035 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.683042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.683045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.683208 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.684494 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.684499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.684502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.684847 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.684974 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.685369 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.685565 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10539,"thread_start_us":10433,"threads_started":1}
I20260926 03:38:23.685948 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.637775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:57953)
0926 03:38:23.647886 (+ 10111us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.647954 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.650649 (+ 2695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.651792 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.651806 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.654259 (+ 2453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.654538 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.654549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.657180 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.657185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.662355 (+ 5170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.662369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.667500 (+ 5131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.667507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.667722 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.669196 (+ 1474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.669218 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.677512 (+ 8294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.682530 (+ 5018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.682534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.682536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.682905 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.684232 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.684237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.684240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.684385 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.684956 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.684961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.685114 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.685255 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.685535 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9751,"thread_start_us":10192,"threads_started":1}
I20260926 03:38:23.686225 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.649431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50883 (local address 127.1.221.188:45897)
0926 03:38:23.652376 (+ 2945us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.652380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.656795 (+ 4415us) server_negotiation.cc:407] Connection header received
0926 03:38:23.656970 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.656975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.657066 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.657161 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.663371 (+ 6210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.664522 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.668074 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.668812 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.668948 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.680077 (+ 11129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.680085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.680091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.680140 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.683351 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.684135 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.684144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.684148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.684309 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.684748 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.684752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.684755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.685067 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.685185 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.685656 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.685876 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2839,"thread_start_us":2756,"threads_started":1}
I20260926 03:38:23.686822 9343 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33115
I20260926 03:38:23.686976 9343 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.687299 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.649319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.1.221.131:50883)
0926 03:38:23.656699 (+ 7380us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.656720 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.656834 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.657345 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.657350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.660560 (+ 3210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:23.660869 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.660881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.665760 (+ 4879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.665765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.667252 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.667266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.670904 (+ 3638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.670911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.671100 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.671825 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:23.671845 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:23.679787 (+ 7942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:23.683530 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.683535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.683539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.683987 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.684455 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:23.684460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:23.684463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:23.684629 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:23.685260 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:23.685265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:23.685400 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.686429 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.686766 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7252,"thread_start_us":7577,"threads_started":1}
I20260926 03:38:23.687537 9343 heartbeater.cc:511] Master 127.1.221.190:33115 requested a full tablet report, sending...
I20260926 03:38:23.688750 9344 heartbeater.cc:347] Connected to a master server at 127.1.221.188:45897
I20260926 03:38:23.688861 9344 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:23.688853 8090 ts_manager.cc:194] Re-registered known tserver with Master: 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:23.689231 9344 heartbeater.cc:511] Master 127.1.221.188:45897 requested a full tablet report, sending...
I20260926 03:38:23.689929 8090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:57953
I20260926 03:38:23.690405 8876 ts_manager.cc:194] Registered new tserver with Master: 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:23.702040 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.658307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.1.221.189:34001)
0926 03:38:23.674674 (+ 16367us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.674681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.681761 (+ 7080us) server_negotiation.cc:407] Connection header received
0926 03:38:23.681943 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.681948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.682034 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.682133 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.684652 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.685703 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.691836 (+ 6133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.692486 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.692590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.695832 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.695836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.695839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.695871 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.698893 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.699820 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.699826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.699830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.699959 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.700394 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.700399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.700402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.700704 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.700940 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.701470 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.701687 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16245,"thread_start_us":20889,"threads_started":1}
I20260926 03:38:23.702950 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.658662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.1.221.190:33115)
0926 03:38:23.680525 (+ 21863us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.680537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.684221 (+ 3684us) server_negotiation.cc:407] Connection header received
0926 03:38:23.684271 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.684276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.684344 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.684437 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.688519 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.689732 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.691521 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.692207 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.692337 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.695836 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.695843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.695847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.695885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.698971 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.700104 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.700109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.700112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.700252 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.700644 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.700649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.700652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.700992 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.701216 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.702353 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.702570 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21083,"thread_start_us":22218,"threads_started":1}
I20260926 03:38:23.707024 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.683344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.1.221.188:45897)
0926 03:38:23.684359 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.684364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.692984 (+ 8620us) server_negotiation.cc:407] Connection header received
0926 03:38:23.693595 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.693600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.693657 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.693729 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:23.695363 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.696366 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.698020 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.698664 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.698792 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.701426 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:23.701434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:23.701439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:23.701477 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:23.704467 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.705181 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.705187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.705191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.705321 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:23.705694 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:23.705698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:23.705701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:23.705956 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:23.706143 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.706445 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.706658 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":710,"threads_started":1}
I20260926 03:38:23.706763 8090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:23.714259 8090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:23.767163 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.732022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43945 (local address 127.0.0.1:58632)
0926 03:38:23.732651 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.732670 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.732790 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.759854 (+ 27064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.759859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.759884 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:23.760382 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.760394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.762430 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.762435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.764552 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.764570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.766348 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.766354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.766512 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.766539 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.766597 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.766697 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":331,"threads_started":1}
I20260926 03:38:23.767935 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.733143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.1.221.130:38159)
0926 03:38:23.757354 (+ 24211us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.757364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.758203 (+ 839us) server_negotiation.cc:407] Connection header received
0926 03:38:23.759714 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.759721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.759963 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.760155 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:23.761676 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.762848 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.765142 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.766053 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.766165 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.766919 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.766989 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.767153 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24031,"thread_start_us":24236,"threads_started":1}
I20260926 03:38:23.767186 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.732473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38159 (local address 127.0.0.1:57764)
0926 03:38:23.758032 (+ 25559us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.758055 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.759500 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.760382 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.760386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.760408 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:23.761493 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.761503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.762975 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.762981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.765016 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.765029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.766250 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.766257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.766435 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.766473 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.766552 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.766695 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25413,"mutex_wait_us":78,"thread_start_us":25196,"threads_started":1}
I20260926 03:38:23.768944 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.732299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.1.221.129:43945)
0926 03:38:23.759347 (+ 27048us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.759355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.759410 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:38:23.759495 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.759502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.759686 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.759878 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:23.761074 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.762302 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.764707 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.766175 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.766298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.768330 (+ 2032us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.768396 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.768567 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26864,"thread_start_us":26730,"threads_started":1}
I20260926 03:38:23.769119 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.735263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.1.221.131:38907)
0926 03:38:23.756699 (+ 21436us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:23.756710 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:23.758778 (+ 2068us) server_negotiation.cc:407] Connection header received
0926 03:38:23.759269 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:23.759282 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:23.759644 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:23.759879 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:23.763018 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.764216 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.766460 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.767795 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.767951 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.768189 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:23.768720 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 03:38:23.768869 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21205,"thread_start_us":21461,"threads_started":1}
I20260926 03:38:23.769181 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:23.734899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:38907 (local address 127.0.0.1:36680)
0926 03:38:23.758649 (+ 23750us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:23.758666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:23.758808 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:23.760598 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:23.760603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:23.760632 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:23.761489 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.761503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.764342 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.764347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.766346 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:23.766359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:23.768378 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:23.768384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:23.768554 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:23.768586 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:38:23.768678 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:38:23.768823 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23638,"mutex_wait_us":18,"thread_start_us":23556,"threads_started":1}
I20260926 03:38:23.772467 9137 tablet_service.cc:1467] Processing CreateTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (DEFAULT_TABLE table=test-table [id=7651bd7f2f0149439a5d684ae583eefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:23.772472 8989 tablet_service.cc:1467] Processing CreateTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (DEFAULT_TABLE table=test-table [id=7651bd7f2f0149439a5d684ae583eefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:23.773648 9137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:23.773736 8989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:23.773761 9278 tablet_service.cc:1467] Processing CreateTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (DEFAULT_TABLE table=test-table [id=7651bd7f2f0149439a5d684ae583eefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:23.774953 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:23.784736 9377 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Bootstrap starting.
I20260926 03:38:23.788257 9377 tablet_bootstrap.cc:654] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:23.789696 9377 log.cc:824] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:23.795096 9378 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Bootstrap starting.
I20260926 03:38:23.799115 9378 tablet_bootstrap.cc:654] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:23.800453 9379 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Bootstrap starting.
I20260926 03:38:23.800524 9378 log.cc:824] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:23.803786 9377 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: No bootstrap required, opened a new log
I20260926 03:38:23.804060 9377 ts_tablet_manager.cc:1397] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20260926 03:38:23.804054 9379 tablet_bootstrap.cc:654] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:23.805655 9378 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: No bootstrap required, opened a new log
I20260926 03:38:23.805781 9379 log.cc:824] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:23.805938 9378 ts_tablet_manager.cc:1397] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:38:23.808796 9379 tablet_bootstrap.cc:492] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: No bootstrap required, opened a new log
I20260926 03:38:23.809039 9379 ts_tablet_manager.cc:1397] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:38:23.812518 9377 raft_consensus.cc:348] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.812986 9377 raft_consensus.cc:374] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:23.813081 9377 raft_consensus.cc:729] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c124cb25afc5489d955b6d46ab759ed4, State: Initialized, Role: FOLLOWER
I20260926 03:38:23.813822 9377 consensus_queue.cc:260] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.814914 9202 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
I20260926 03:38:23.815665 9377 ts_tablet_manager.cc:1428] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 03:38:23.816099 9378 raft_consensus.cc:348] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.816519 9378 raft_consensus.cc:374] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:23.816597 9378 raft_consensus.cc:729] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f314af49ade4796a5aeba5fe0caa7d7, State: Initialized, Role: FOLLOWER
I20260926 03:38:23.817201 9378 consensus_queue.cc:260] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.819242 9379 raft_consensus.cc:348] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.819748 9379 raft_consensus.cc:374] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:23.819842 9379 raft_consensus.cc:729] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f159911fb14d44bf7ba4d09ccc6f1e, State: Initialized, Role: FOLLOWER
I20260926 03:38:23.820508 9379 consensus_queue.cc:260] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:23.821152 9378 ts_tablet_manager.cc:1428] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260926 03:38:23.826297 9343 heartbeater.cc:503] Master 127.1.221.190:33115 was elected leader, sending a full tablet report...
I20260926 03:38:23.828554 9379 ts_tablet_manager.cc:1428] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
W20260926 03:38:23.887084 9346 tablet.cc:2369] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:38:23.967092 9057 tablet.cc:2369] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:38:24.048732 9205 tablet.cc:2369] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:38:24.209935 9384 raft_consensus.cc:482] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:24.210276 9384 raft_consensus.cc:504] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:24.212103 9384 leader_election.cc:288] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159), 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:24.219288 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.212634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34815 (local address 127.1.221.131:38907)
0926 03:38:24.212847 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.212853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.213422 (+ 569us) server_negotiation.cc:407] Connection header received
0926 03:38:24.213583 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.213587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.213646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.213736 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:24.214328 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.215030 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.217031 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.218224 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.218408 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.218483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.218877 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.219011 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:38:24.219290 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.212429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:50781 (local address 127.1.221.130:38159)
0926 03:38:24.212682 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.212689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.213304 (+ 615us) server_negotiation.cc:407] Connection header received
0926 03:38:24.213528 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.213534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.213605 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.213671 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:24.214330 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.215035 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.217112 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.218283 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.218414 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.218483 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.218877 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.219024 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:38:24.219523 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.212358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38159 (local address 127.1.221.129:50781)
0926 03:38:24.213183 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:24.213199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:24.213352 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:24.213710 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:24.213715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:24.213738 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:24.214184 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:24.214193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.215165 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.215170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:24.216909 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:24.216922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.218472 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.218478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:24.218677 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.218757 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 03:38:24.218845 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:38:24.218994 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":104,"thread_start_us":490,"threads_started":1}
I20260926 03:38:24.219597 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.212576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:38907 (local address 127.1.221.129:34815)
0926 03:38:24.213318 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:24.213332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:24.213452 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:24.213765 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:24.213769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:24.213791 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:24.214177 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:24.214187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.215165 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.215169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:24.216884 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:24.216896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.218442 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.218450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:24.218660 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.218751 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 03:38:24.218845 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:38:24.218994 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":84,"thread_start_us":483,"threads_started":1}
I20260926 03:38:24.220538 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa76e3e5869540a7ba0dd8b7bb6ddba2" candidate_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124cb25afc5489d955b6d46ab759ed4" is_pre_election: true
I20260926 03:38:24.221073 9157 raft_consensus.cc:2454] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f314af49ade4796a5aeba5fe0caa7d7 in term 0.
I20260926 03:38:24.221243 9298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa76e3e5869540a7ba0dd8b7bb6ddba2" candidate_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" is_pre_election: true
I20260926 03:38:24.221737 9298 raft_consensus.cc:2454] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f314af49ade4796a5aeba5fe0caa7d7 in term 0.
I20260926 03:38:24.221800 8945 leader_election.cc:302] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f314af49ade4796a5aeba5fe0caa7d7, c124cb25afc5489d955b6d46ab759ed4; no voters:
I20260926 03:38:24.222261 9384 raft_consensus.cc:2781] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:24.222409 9384 raft_consensus.cc:482] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:24.222510 9384 raft_consensus.cc:3037] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:24.224803 9384 raft_consensus.cc:504] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:24.225389 9384 leader_election.cc:288] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [CANDIDATE]: Term 1 election: Requested vote from peers c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159), 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907)
I20260926 03:38:24.225924 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa76e3e5869540a7ba0dd8b7bb6ddba2" candidate_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124cb25afc5489d955b6d46ab759ed4"
I20260926 03:38:24.225939 9298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa76e3e5869540a7ba0dd8b7bb6ddba2" candidate_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e"
I20260926 03:38:24.226133 9157 raft_consensus.cc:3037] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:24.226151 9298 raft_consensus.cc:3037] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:24.228543 9298 raft_consensus.cc:2454] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f314af49ade4796a5aeba5fe0caa7d7 in term 1.
I20260926 03:38:24.229000 9157 raft_consensus.cc:2454] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f314af49ade4796a5aeba5fe0caa7d7 in term 1.
I20260926 03:38:24.228991 8943 leader_election.cc:302] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f159911fb14d44bf7ba4d09ccc6f1e, 3f314af49ade4796a5aeba5fe0caa7d7; no voters:
I20260926 03:38:24.229574 9384 raft_consensus.cc:2781] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:24.230005 9384 raft_consensus.cc:686] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f314af49ade4796a5aeba5fe0caa7d7, State: Running, Role: LEADER
I20260926 03:38:24.230801 9384 consensus_queue.cc:237] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:24.237179 8089 catalog_manager.cc:5495] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } health_report { overall_health: UNKNOWN } } }
I20260926 03:38:24.341272 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.334788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.1.221.129:43945)
0926 03:38:24.335722 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.335727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.335743 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:38:24.335831 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.335835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.335909 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.336052 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:24.336482 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.337201 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.339117 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.339574 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.339669 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.340202 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:24.340726 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:24.340839 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.340938 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.341013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":730,"threads_started":1}
I20260926 03:38:24.348925 9157 raft_consensus.cc:1261] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Refusing update from remote peer 3f314af49ade4796a5aeba5fe0caa7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:24.348924 9298 raft_consensus.cc:1261] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Refusing update from remote peer 3f314af49ade4796a5aeba5fe0caa7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:24.349864 9388 consensus_queue.cc:1035] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:24.350239 9384 consensus_queue.cc:1035] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:24.362062 9401 mvcc.cc:205] Tried to move back new op lower bound from 7333453432203931648 to 7333453431733280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333453432203931648}]
I20260926 03:38:24.365523 9404 mvcc.cc:205] Tried to move back new op lower bound from 7333453432203931648 to 7333453431733280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333453432203931648}]
I20260926 03:38:24.382511 8089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:24.382758 8089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:24.388566 8089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac94d102a7f647b3a8cc7772a73ff2d8: Sending DeleteTablet for 3 replicas of tablet aa76e3e5869540a7ba0dd8b7bb6ddba2
I20260926 03:38:24.390064 9137 tablet_service.cc:1514] Processing DeleteTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:24 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57764
I20260926 03:38:24.390664 8989 tablet_service.cc:1514] Processing DeleteTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:24 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:58632
I20260926 03:38:24.391103 9278 tablet_service.cc:1514] Processing DeleteTablet for tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:24 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:36680
I20260926 03:38:24.401719 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8058
I20260926 03:38:24.402961 8120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:24.414741 9412 tablet_replica.cc:330] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: stopping tablet replica
I20260926 03:38:24.414921 9414 tablet_replica.cc:330] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: stopping tablet replica
I20260926 03:38:24.415457 9414 raft_consensus.cc:2229] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:24.416019 9414 raft_consensus.cc:2258] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:24.415997 9412 raft_consensus.cc:2229] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:24.416581 9412 raft_consensus.cc:2258] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:24.418562 9412 ts_tablet_manager.cc:1905] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:24.418565 9414 ts_tablet_manager.cc:1905] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:24.426347 9412 ts_tablet_manager.cc:1918] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:24.426638 9412 log.cc:1196] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/aa76e3e5869540a7ba0dd8b7bb6ddba2
I20260926 03:38:24.427285 9412 ts_tablet_manager.cc:1939] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P c124cb25afc5489d955b6d46ab759ed4: Deleting consensus metadata
I20260926 03:38:24.431787 9414 ts_tablet_manager.cc:1918] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:24.416836 9413 tablet_replica.cc:330] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: stopping tablet replica
I20260926 03:38:24.432562 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.393323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.1.221.189:34001)
0926 03:38:24.399721 (+ 6398us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.399727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.399745 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:24.399806 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.399811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.399886 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.399989 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:24.415609 (+ 15620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.417873 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.419963 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.420614 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.420740 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.424507 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:24.424514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:24.424519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:24.424618 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:24.428302 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.429338 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.429344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.429348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.429512 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.429966 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.429971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.429974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.430308 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:24.430562 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.431939 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.432188 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6280,"thread_start_us":6233,"threads_started":1}
I20260926 03:38:24.432538 9413 raft_consensus.cc:2229] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:24.433483 9413 raft_consensus.cc:2258] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:24.435655 9413 ts_tablet_manager.cc:1905] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:24.437813 9414 log.cc:1196] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/aa76e3e5869540a7ba0dd8b7bb6ddba2
I20260926 03:38:24.438345 9414 ts_tablet_manager.cc:1939] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting consensus metadata
I20260926 03:38:24.446178 9413 ts_tablet_manager.cc:1918] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:24.446727 9413 log.cc:1196] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/aa76e3e5869540a7ba0dd8b7bb6ddba2
I20260926 03:38:24.447355 9413 ts_tablet_manager.cc:1939] T aa76e3e5869540a7ba0dd8b7bb6ddba2 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting consensus metadata
I20260926 03:38:24.449795 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.399422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.1.221.188:45897)
0926 03:38:24.414325 (+ 14903us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.414332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.433210 (+ 18878us) server_negotiation.cc:407] Connection header received
0926 03:38:24.433287 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.433293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.433380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.433501 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:24.435167 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.436358 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.438174 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.438859 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.438980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.442237 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:24.442244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:24.442248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:24.442292 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:24.445816 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.447345 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.447351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.447355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.447624 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.448109 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.448114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.448116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.448468 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:24.448712 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.449361 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.449548 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14742,"thread_start_us":14636,"threads_started":1}
I20260926 03:38:24.576056 8077 catalog_manager.cc:4841] TS c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159): tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (table test-table [id=7651bd7f2f0149439a5d684ae583eefd]) successfully deleted
I20260926 03:38:24.576150 8075 catalog_manager.cc:4841] TS 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945): tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (table test-table [id=7651bd7f2f0149439a5d684ae583eefd]) successfully deleted
I20260926 03:38:24.576547 8075 catalog_manager.cc:4841] TS 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907): tablet aa76e3e5869540a7ba0dd8b7bb6ddba2 (table test-table [id=7651bd7f2f0149439a5d684ae583eefd]) successfully deleted
I20260926 03:38:24.576720 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8058
I20260926 03:38:24.615460 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.600507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.1.221.188:45897)
0926 03:38:24.600673 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.600677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.600692 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:38:24.600831 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.600837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.600910 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.601012 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:24.603119 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.604063 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.605456 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.606130 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.606251 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.608707 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:24.608712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:24.608715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:24.608747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:24.611925 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.612774 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.612780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.612784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.612951 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.613339 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.613345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.613348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.613710 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:24.613938 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.614825 (+ 887us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.615075 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:38:24.615547 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:24.598649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.1.221.189:34001)
0926 03:38:24.598841 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:24.598848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:24.599903 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 03:38:24.600270 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:24.600276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:24.600364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:24.600470 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:24.602328 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.603167 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.604993 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:24.605599 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:24.605717 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:24.609715 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:24.609721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:24.609725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:24.609770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:24.612751 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.613482 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.613487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.613490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.613629 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:24.614007 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:24.614010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:24.614012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:24.614265 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:24.614450 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:24.614888 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 03:38:24.615098 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:38:25.374164 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.373311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:42321)
0926 03:38:25.373792 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.373955 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
W20260926 03:38:25.374795 9202 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:33115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.376046 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.374437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:58367)
0926 03:38:25.375739 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.375873 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1167,"threads_started":1}
I20260926 03:38:25.376235 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.374245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:59991)
0926 03:38:25.374846 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.375138 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":363,"threads_started":1}
W20260926 03:38:25.376677 9054 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:33115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
W20260926 03:38:25.377449 9343 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:33115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.889230 9428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac94d102a7f647b3a8cc7772a73ff2d8)
I20260926 03:38:25.889559 9428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } }
I20260926 03:38:25.890100 9429 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac94d102a7f647b3a8cc7772a73ff2d8)
I20260926 03:38:25.890372 9429 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } }
I20260926 03:38:25.891165 9428 leader_election.cc:288] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115), a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189:34001)
I20260926 03:38:25.893690 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.891450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38514)
0926 03:38:25.893300 (+ 1850us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.893461 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1728,"thread_start_us":1603,"threads_started":1}
W20260926 03:38:25.893947 8862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.896220 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.895229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38540)
0926 03:38:25.895758 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.895922 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260926 03:38:25.896489 8862 leader_election.cc:334] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115): Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.900167 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.891433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46790)
0926 03:38:25.891926 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.891946 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:25.892085 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:25.892813 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:25.892817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:25.892845 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:25.893186 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.893199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.894918 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.894925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:25.897397 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.897415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.899300 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.899306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:25.899509 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:25.899544 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:38:25.899628 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:38:25.899771 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":359,"threads_started":1}
I20260926 03:38:25.900875 9429 leader_election.cc:288] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115), 893121ecd9a9439294f32606274e8c92 (127.1.221.188:45897)
I20260926 03:38:25.901407 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.891666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.1.221.189:34001)
0926 03:38:25.892551 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:25.892555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:25.892570 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:38:25.892632 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:25.892635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:25.892689 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:25.892771 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:25.893446 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.894726 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.897610 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.898947 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.899079 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:25.899200 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:25.900485 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0926 03:38:25.900651 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":612,"threads_started":1}
I20260926 03:38:25.902153 8697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "893121ecd9a9439294f32606274e8c92" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" is_pre_election: true
I20260926 03:38:25.902554 8697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 893121ecd9a9439294f32606274e8c92 in term 1.
I20260926 03:38:25.903280 8859 leader_election.cc:302] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 893121ecd9a9439294f32606274e8c92, a9a2385f1d0d432e85c11f12e4187c3c; no voters: ac94d102a7f647b3a8cc7772a73ff2d8
I20260926 03:38:25.903940 9428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:38:25.904073 9428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac94d102a7f647b3a8cc7772a73ff2d8)
I20260926 03:38:25.904166 9428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:38:25.904389 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.891862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38524)
0926 03:38:25.902593 (+ 10731us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.904106 (+ 1513us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10579,"thread_start_us":11906,"threads_started":1}
W20260926 03:38:25.904686 8674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.906733 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.906304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38558)
0926 03:38:25.906468 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.906583 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:38:25.906941 8674 leader_election.cc:334] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115): Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.907301 9428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } }
I20260926 03:38:25.908370 8697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "893121ecd9a9439294f32606274e8c92" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9a2385f1d0d432e85c11f12e4187c3c"
I20260926 03:38:25.908494 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.908051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38568)
0926 03:38:25.908257 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.908347 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 03:38:25.908639 8697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:38:25.908921 9428 leader_election.cc:288] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 election: Requested vote from peers ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115), a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189:34001)
I20260926 03:38:25.909513 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.909129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38584)
0926 03:38:25.909284 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.909365 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:38:25.909747 8862 leader_election.cc:334] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115): Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
I20260926 03:38:25.911005 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.896865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.1.221.188:45897)
0926 03:38:25.904807 (+ 7942us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:25.904813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:25.904835 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:38:25.904889 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:25.904894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:25.904970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:25.905079 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:25.905807 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.906758 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.908980 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.910083 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.910196 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:25.910296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:25.910573 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:38:25.910707 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7811,"thread_start_us":7650,"threads_started":1}
I20260926 03:38:25.910992 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.894914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45897 (local address 127.0.0.1:45076)
0926 03:38:25.903160 (+ 8246us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.903179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:25.903310 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:25.905132 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:25.905137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:25.905172 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:25.905616 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.905626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.906906 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.906909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:25.908810 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.908826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.910255 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.910261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:25.910432 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:25.910458 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:38:25.910533 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:38:25.910669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8124,"thread_start_us":7373,"threads_started":1}
I20260926 03:38:25.911036 8697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 893121ecd9a9439294f32606274e8c92 in term 2.
I20260926 03:38:25.911470 8859 leader_election.cc:302] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 893121ecd9a9439294f32606274e8c92, a9a2385f1d0d432e85c11f12e4187c3c; no voters: ac94d102a7f647b3a8cc7772a73ff2d8
I20260926 03:38:25.911644 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "893121ecd9a9439294f32606274e8c92" is_pre_election: true
I20260926 03:38:25.911774 9428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:38:25.912173 9428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [term 2 LEADER]: Becoming Leader. State: Replica: 893121ecd9a9439294f32606274e8c92, State: Running, Role: LEADER
I20260926 03:38:25.912508 9428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } }
I20260926 03:38:25.914515 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893121ecd9a9439294f32606274e8c92. Latest consensus state: current_term: 2 leader_uuid: "893121ecd9a9439294f32606274e8c92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:25.914919 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:25.914945 8671 leader_election.cc:302] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9a2385f1d0d432e85c11f12e4187c3c; no voters: 893121ecd9a9439294f32606274e8c92, ac94d102a7f647b3a8cc7772a73ff2d8
I20260926 03:38:25.915282 9429 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:38:25.915587 9440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:25.918896 9440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:25.919720 9440 catalog_manager.cc:1259] Loaded cluster ID: 1188976a5f3e459a84b373551db5db82
I20260926 03:38:25.919791 9440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:25.920863 9440 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:25.921599 9440 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Loaded TSK: 0
I20260926 03:38:25.922402 9440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:25.966496 8876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:25.973768 8876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:25.978055 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.977649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38606)
0926 03:38:25.977801 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.977899 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:38:25.978241 8697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [term 2 FOLLOWER]: Refusing update from remote peer 893121ecd9a9439294f32606274e8c92: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:38:25.979169 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.978675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38620)
0926 03:38:25.978871 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.978977 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:38:25.979142 9438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 03:38:25.979539 8862 consensus_peers.cc:487] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 -> Peer ac94d102a7f647b3a8cc7772a73ff2d8 (127.1.221.190:33115): Couldn't send request to peer ac94d102a7f647b3a8cc7772a73ff2d8. Status: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:38:25.982712 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893121ecd9a9439294f32606274e8c92. Latest consensus state: current_term: 2 leader_uuid: "893121ecd9a9439294f32606274e8c92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:25.982998 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:25.984025 9428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "893121ecd9a9439294f32606274e8c92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:25.984431 9428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:25.984476 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "893121ecd9a9439294f32606274e8c92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac94d102a7f647b3a8cc7772a73ff2d8" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33115 } } peers { permanent_uuid: "a9a2385f1d0d432e85c11f12e4187c3c" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34001 } attrs { promote: false } } peers { permanent_uuid: "893121ecd9a9439294f32606274e8c92" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45897 } attrs { promote: false } } }
I20260926 03:38:25.984754 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2385f1d0d432e85c11f12e4187c3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:38:25.984858 9442 mvcc.cc:205] Tried to move back new op lower bound from 7333453438879248384 to 7333453438621007872. Current Snapshot: MvccSnapshot[applied={T|T < 7333453438879248384}]
I20260926 03:38:26.000715 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.992596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38159 (local address 127.0.0.1:57776)
0926 03:38:25.992806 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.992825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:25.992935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:25.993560 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:25.993566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:25.993599 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:25.993824 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.993835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.996277 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.996284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:25.998254 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:25.998270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.999896 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.999903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.000137 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.000180 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:38:26.000264 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:38:26.000410 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:38:26.002302 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.992737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.1.221.130:38159)
0926 03:38:25.993210 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:25.993216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:25.993235 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:25.993311 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:25.993316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:25.993393 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:25.993965 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:25.994147 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.996096 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:25.998470 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:25.999723 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.001666 (+ 1943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.001792 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.001849 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.001997 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260926 03:38:26.003347 9137 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 (DEFAULT_TABLE table=test-table [id=49ac954a4a21408b8758aff880874513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:26.003867 9137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec2875c1e9b4480a4926e695305ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:26.009397 9451 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Bootstrap starting.
I20260926 03:38:26.010334 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.993673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43945 (local address 127.0.0.1:58650)
0926 03:38:25.993977 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:25.993999 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:25.994108 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:26.004102 (+ 9994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:26.004107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:26.004137 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:26.004423 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.004434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.005983 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.005989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.007932 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.007947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.009542 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.009549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.009736 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.009776 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:38:26.009848 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:38:26.010008 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 03:38:26.011298 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.994339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.1.221.129:43945)
0926 03:38:26.003759 (+ 9420us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.003765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.003786 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:38:26.003856 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.003861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.003941 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.004045 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:26.004564 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.005838 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.008074 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.009275 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.010642 (+ 1367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.010752 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.010807 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.010954 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9290,"thread_start_us":9188,"threads_started":1}
I20260926 03:38:26.011695 9451 tablet_bootstrap.cc:654] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:26.012799 8989 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 (DEFAULT_TABLE table=test-table [id=49ac954a4a21408b8758aff880874513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:26.013293 8989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec2875c1e9b4480a4926e695305ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:26.014084 9451 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: No bootstrap required, opened a new log
I20260926 03:38:26.014225 9451 ts_tablet_manager.cc:1397] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 03:38:26.014868 9451 raft_consensus.cc:348] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.015079 9451 raft_consensus.cc:374] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:26.015182 9451 raft_consensus.cc:729] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c124cb25afc5489d955b6d46ab759ed4, State: Initialized, Role: FOLLOWER
I20260926 03:38:26.015455 9451 consensus_queue.cc:260] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.017896 9451 ts_tablet_manager.cc:1428] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260926 03:38:26.019479 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.018263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:53829)
0926 03:38:26.018985 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.019193 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"threads_started":1}
I20260926 03:38:26.020762 9455 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Bootstrap starting.
I20260926 03:38:26.022573 9455 tablet_bootstrap.cc:654] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:26.022567 9203 heartbeater.cc:503] Master 127.1.221.188:45897 was elected leader, sending a full tablet report...
I20260926 03:38:26.026096 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.993980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:38907 (local address 127.0.0.1:36690)
0926 03:38:26.000956 (+ 6976us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.000976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:26.001100 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:26.016160 (+ 15060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:26.016168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:26.016210 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:26.016507 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.016519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.021728 (+ 5209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.021735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.023639 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.023659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.025162 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.025169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.025429 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.025469 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:38:26.025578 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:38:26.025737 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6878,"thread_start_us":17902,"threads_started":1}
I20260926 03:38:26.026362 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.023146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:57895)
0926 03:38:26.024662 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.026215 (+ 1553us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1388}
I20260926 03:38:26.027197 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:25.994401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.1.221.131:38907)
0926 03:38:26.015764 (+ 21363us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.015770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.015794 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:38:26.015869 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.015874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.015962 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.016102 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:26.020549 (+ 4447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.021573 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.023838 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.024990 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.025155 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.025265 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.026698 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.026881 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21236,"thread_start_us":21133,"threads_started":1}
I20260926 03:38:26.028067 9278 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 (DEFAULT_TABLE table=test-table [id=49ac954a4a21408b8758aff880874513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:26.028509 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec2875c1e9b4480a4926e695305ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:26.028808 9455 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: No bootstrap required, opened a new log
I20260926 03:38:26.028968 9455 ts_tablet_manager.cc:1397] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260926 03:38:26.029784 9455 raft_consensus.cc:348] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.030045 9455 raft_consensus.cc:374] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:26.030153 9455 raft_consensus.cc:729] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f314af49ade4796a5aeba5fe0caa7d7, State: Initialized, Role: FOLLOWER
I20260926 03:38:26.030366 9455 consensus_queue.cc:260] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.031126 9455 ts_tablet_manager.cc:1428] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:38:26.033373 9055 heartbeater.cc:503] Master 127.1.221.188:45897 was elected leader, sending a full tablet report...
I20260926 03:38:26.034989 9459 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Bootstrap starting.
I20260926 03:38:26.036273 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.031411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:59835)
0926 03:38:26.035833 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.036008 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4293,"thread_start_us":4165,"threads_started":1}
I20260926 03:38:26.036772 9459 tablet_bootstrap.cc:654] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:26.038604 9459 tablet_bootstrap.cc:492] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: No bootstrap required, opened a new log
I20260926 03:38:26.038739 9459 ts_tablet_manager.cc:1397] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:38:26.039433 9459 raft_consensus.cc:348] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.039638 9459 raft_consensus.cc:374] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:26.039721 9459 raft_consensus.cc:729] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f159911fb14d44bf7ba4d09ccc6f1e, State: Initialized, Role: FOLLOWER
I20260926 03:38:26.039927 9459 consensus_queue.cc:260] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.040732 9459 ts_tablet_manager.cc:1428] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:38:26.044368 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.043987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:48769)
0926 03:38:26.044151 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.044232 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:38:26.048738 9344 heartbeater.cc:503] Master 127.1.221.188:45897 was elected leader, sending a full tablet report...
I20260926 03:38:26.049566 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.040965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:55931)
0926 03:38:26.049086 (+ 8121us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.049274 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7978,"thread_start_us":7829,"threads_started":1}
I20260926 03:38:26.050797 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.050382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:44877)
0926 03:38:26.050594 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.050674 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:38:26.197706 9461 raft_consensus.cc:482] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:26.197981 9461 raft_consensus.cc:504] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.199687 9461 leader_election.cc:288] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159), 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:26.205471 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.199961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38159 (local address 127.1.221.131:58077)
0926 03:38:26.200150 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.200170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:26.200338 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:26.200669 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:26.200674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:26.200705 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:26.200967 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.200976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.201945 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.201949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.203465 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.203478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.204835 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.204840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.204984 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.205010 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:38:26.205088 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:38:26.205204 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:38:26.205518 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.200129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:58077 (local address 127.1.221.130:38159)
0926 03:38:26.200326 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.200331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.200353 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:38:26.200485 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.200489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.200548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.200626 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:26.201098 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.201815 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.203619 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.204656 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.204810 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.204887 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.205129 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.205256 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:38:26.206147 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.200100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59393 (local address 127.1.221.129:43945)
0926 03:38:26.200298 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.200304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.200946 (+ 642us) server_negotiation.cc:407] Connection header received
0926 03:38:26.201111 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.201115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.201184 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.201257 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:26.201727 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.202405 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.204134 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.205230 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.205379 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.205448 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.205823 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.205941 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:38:26.206116 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec2875c1e9b4480a4926e695305ad34" candidate_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124cb25afc5489d955b6d46ab759ed4" is_pre_election: true
I20260926 03:38:26.206377 9157 raft_consensus.cc:2454] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f159911fb14d44bf7ba4d09ccc6f1e in term 0.
I20260926 03:38:26.206813 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.199961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43945 (local address 127.1.221.131:59393)
0926 03:38:26.200840 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.200854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:26.200985 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:26.201281 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:26.201285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:26.201307 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:26.201595 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.201604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.202528 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.202533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.203995 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:26.204006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.205442 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.205448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:26.205652 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.205683 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:38:26.206309 (+ 626us) client_negotiation.cc:240] Negotiation successful
0926 03:38:26.206442 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":227,"thread_start_us":384,"threads_started":1}
I20260926 03:38:26.206916 9234 leader_election.cc:302] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f159911fb14d44bf7ba4d09ccc6f1e, c124cb25afc5489d955b6d46ab759ed4; no voters:
I20260926 03:38:26.207311 9461 raft_consensus.cc:2781] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:26.207475 9461 raft_consensus.cc:482] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:26.207574 9461 raft_consensus.cc:3037] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:26.207674 9009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec2875c1e9b4480a4926e695305ad34" candidate_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" is_pre_election: true
I20260926 03:38:26.208106 9009 raft_consensus.cc:2454] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f159911fb14d44bf7ba4d09ccc6f1e in term 0.
I20260926 03:38:26.211452 9461 raft_consensus.cc:504] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.212059 9461 leader_election.cc:288] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [CANDIDATE]: Term 1 election: Requested vote from peers c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159), 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945)
I20260926 03:38:26.212555 9009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec2875c1e9b4480a4926e695305ad34" candidate_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f314af49ade4796a5aeba5fe0caa7d7"
I20260926 03:38:26.212546 9157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec2875c1e9b4480a4926e695305ad34" candidate_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c124cb25afc5489d955b6d46ab759ed4"
I20260926 03:38:26.212729 9009 raft_consensus.cc:3037] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:26.212735 9157 raft_consensus.cc:3037] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:26.214656 9157 raft_consensus.cc:2454] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f159911fb14d44bf7ba4d09ccc6f1e in term 1.
I20260926 03:38:26.214665 9009 raft_consensus.cc:2454] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f159911fb14d44bf7ba4d09ccc6f1e in term 1.
I20260926 03:38:26.215051 9232 leader_election.cc:302] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f159911fb14d44bf7ba4d09ccc6f1e, 3f314af49ade4796a5aeba5fe0caa7d7; no voters:
I20260926 03:38:26.215356 9461 raft_consensus.cc:2781] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:26.215793 9461 raft_consensus.cc:686] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 LEADER]: Becoming Leader. State: Replica: 22f159911fb14d44bf7ba4d09ccc6f1e, State: Running, Role: LEADER
I20260926 03:38:26.216182 9461 consensus_queue.cc:237] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } }
I20260926 03:38:26.216794 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.216409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:50441)
0926 03:38:26.216577 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.216665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:38:26.219578 8876 catalog_manager.cc:5495] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131). New cstate: current_term: 1 leader_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22f159911fb14d44bf7ba4d09ccc6f1e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 38907 } health_report { overall_health: HEALTHY } } }
I20260926 03:38:26.265617 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.260732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.1.221.131:38907)
0926 03:38:26.260909 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.260914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.260934 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:38:26.261059 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.261063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.261147 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.261272 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:26.261645 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.262330 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.263630 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.264023 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.264132 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.264489 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:26.265046 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:26.265174 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.265270 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.265353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:38:26.269572 9157 raft_consensus.cc:1261] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Refusing update from remote peer 22f159911fb14d44bf7ba4d09ccc6f1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:26.269906 9009 raft_consensus.cc:1261] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Refusing update from remote peer 22f159911fb14d44bf7ba4d09ccc6f1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:26.270423 9464 consensus_queue.cc:1035] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c124cb25afc5489d955b6d46ab759ed4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:26.270829 9461 consensus_queue.cc:1035] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f314af49ade4796a5aeba5fe0caa7d7" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:26.273361 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.272849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:50715)
0926 03:38:26.273061 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.273175 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 03:38:26.273674 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.272844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:54573)
0926 03:38:26.273401 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.273492 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260926 03:38:26.279026 9467 mvcc.cc:205] Tried to move back new op lower bound from 7333453440073400320 to 7333453439865569280. Current Snapshot: MvccSnapshot[applied={T|T < 7333453440073400320}]
I20260926 03:38:26.279858 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.279298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.130:45943)
0926 03:38:26.279594 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.279688 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 03:38:26.285567 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.285083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.131:45789)
0926 03:38:26.285320 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.285417 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 03:38:26.288180 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.287683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.1.221.129:37151)
0926 03:38:26.287923 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.288012 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 03:38:26.295756 8875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:26.296038 8875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:26.302119 8875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92: Sending DeleteTablet for 3 replicas of tablet 8ec2875c1e9b4480a4926e695305ad34
I20260926 03:38:26.303120 9137 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:26 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:57776
I20260926 03:38:26.303233 8989 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:26 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:58650
I20260926 03:38:26.303550 9278 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec2875c1e9b4480a4926e695305ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:26 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:36690
I20260926 03:38:26.311368 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8863
I20260926 03:38:26.315618 9479 tablet_replica.cc:330] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: stopping tablet replica
I20260926 03:38:26.315632 9478 tablet_replica.cc:330] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: stopping tablet replica
I20260926 03:38:26.315842 9478 raft_consensus.cc:2229] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:26.315997 9479 raft_consensus.cc:2229] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:26.316293 9478 raft_consensus.cc:2258] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:26.316325 9479 raft_consensus.cc:2258] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:26.317420 9479 ts_tablet_manager.cc:1905] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:26.317895 9478 ts_tablet_manager.cc:1905] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:26.324043 9478 ts_tablet_manager.cc:1918] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:26.324043 9479 ts_tablet_manager.cc:1918] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:26.324177 9479 log.cc:1196] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/8ec2875c1e9b4480a4926e695305ad34
I20260926 03:38:26.324285 9478 log.cc:1196] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/8ec2875c1e9b4480a4926e695305ad34
I20260926 03:38:26.324685 9479 ts_tablet_manager.cc:1939] T 8ec2875c1e9b4480a4926e695305ad34 P 3f314af49ade4796a5aeba5fe0caa7d7: Deleting consensus metadata
I20260926 03:38:26.324739 9478 ts_tablet_manager.cc:1939] T 8ec2875c1e9b4480a4926e695305ad34 P c124cb25afc5489d955b6d46ab759ed4: Deleting consensus metadata
I20260926 03:38:26.325907 8860 catalog_manager.cc:4841] TS 3f314af49ade4796a5aeba5fe0caa7d7 (127.1.221.129:43945): tablet 8ec2875c1e9b4480a4926e695305ad34 (table test-table [id=49ac954a4a21408b8758aff880874513]) successfully deleted
I20260926 03:38:26.325960 8862 catalog_manager.cc:4841] TS c124cb25afc5489d955b6d46ab759ed4 (127.1.221.130:38159): tablet 8ec2875c1e9b4480a4926e695305ad34 (table test-table [id=49ac954a4a21408b8758aff880874513]) successfully deleted
I20260926 03:38:26.328459 9480 tablet_replica.cc:330] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: stopping tablet replica
I20260926 03:38:26.328665 9480 raft_consensus.cc:2229] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:26.329219 9480 raft_consensus.cc:2258] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:26.333427 9480 ts_tablet_manager.cc:1905] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:26.339823 9480 ts_tablet_manager.cc:1918] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:26.340071 9480 log.cc:1196] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/8ec2875c1e9b4480a4926e695305ad34
I20260926 03:38:26.340622 9480 ts_tablet_manager.cc:1939] T 8ec2875c1e9b4480a4926e695305ad34 P 22f159911fb14d44bf7ba4d09ccc6f1e: Deleting consensus metadata
I20260926 03:38:26.341910 8860 catalog_manager.cc:4841] TS 22f159911fb14d44bf7ba4d09ccc6f1e (127.1.221.131:38907): tablet 8ec2875c1e9b4480a4926e695305ad34 (table test-table [id=49ac954a4a21408b8758aff880874513]) successfully deleted
I20260926 03:38:26.345331 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.312802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.1.221.129:43945)
0926 03:38:26.312992 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.312998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.327895 (+ 14897us) server_negotiation.cc:407] Connection header received
0926 03:38:26.328062 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.328067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.328160 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.328253 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:26.330721 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.331508 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.332808 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.333349 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.333472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.338835 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:26.338856 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:26.338878 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:26.338927 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:26.342619 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.343311 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.343318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.343321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.343491 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.343861 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.343867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.343870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.344224 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:26.344499 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.344795 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.344996 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:38:26.346055 9049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:26.466969 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8863
I20260926 03:38:26.493791 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9066
I20260926 03:38:26.502496 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.502049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38628)
0926 03:38:26.502229 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.502335 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:38:26.503844 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.503456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38634)
0926 03:38:26.503601 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.503700 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 03:38:26.507629 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.494524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.1.221.130:38159)
0926 03:38:26.494703 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.494709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.494787 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:38:26.494962 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.494968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.495053 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.495146 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:26.496459 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.497214 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.498436 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.498874 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.498986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.501588 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:26.501604 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:26.501622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:26.501660 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:26.504928 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.505578 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.505586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.505589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.505729 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.506084 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.506089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.506092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.506450 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:26.506725 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.507037 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.507255 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:38:26.508265 9197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:26.601083 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9066
I20260926 03:38:26.621363 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9215
I20260926 03:38:26.638222 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.622104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.1.221.131:38907)
0926 03:38:26.622296 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:26.622300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:26.622351 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:38:26.622549 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:26.622553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:26.622637 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:26.622722 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:26.624630 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.625421 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.627216 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:26.627810 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:26.627907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:26.631120 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:26.631140 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:26.631160 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:26.631201 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:26.634936 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.635902 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.635912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.635916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.636092 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:26.636530 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:26.636536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:26.636539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:26.637022 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:26.637353 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:26.637640 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:38:26.637860 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:38:26.638934 9338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:26.765231 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9215
I20260926 03:38:26.783114 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8551
I20260926 03:38:26.784188 8717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:26.881765 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.881337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38650)
0926 03:38:26.881503 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.881610 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:38:26.882756 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.882438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33115 (local address 127.0.0.1:38652)
0926 03:38:26.882565 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.882639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 03:38:26.898378 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8551
W20260926 03:38:26.914415 8859 connection.cc:535] client connection to 127.1.221.189:34001 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:38:26.914712 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 8650
I20260926 03:38:26.915551 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:26.915127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34001 (local address 127.0.0.1:46794)
0926 03:38:26.915282 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:26.915357 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:38:26.915756 8906 generic_service.cc:196] Checking for leaks (request via RPC)
W20260926 03:38:26.915763 8859 consensus_peers.cc:487] T 00000000000000000000000000000000 P 893121ecd9a9439294f32606274e8c92 -> Peer a9a2385f1d0d432e85c11f12e4187c3c (127.1.221.189:34001): Couldn't send request to peer a9a2385f1d0d432e85c11f12e4187c3c. Status: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:38:27.033254 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 8650
2026-09-26T03:38:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9662 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:27 dist-test-slave-lqrk krb5kdc[9490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:27 dist-test-slave-lqrk krb5kdc[9490](info): set up 2 sockets
Sep 26 03:38:27 dist-test-slave-lqrk krb5kdc[9490](info): commencing operation
krb5kdc: starting...
W20260926 03:38:28.036204 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:28 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
I20260926 03:38:28.247372 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36063
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:28.573249 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:28.573558 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:28.573690 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:28.581058 9503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:28.581168 9503 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:38:28.581230 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:28.581312 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:28.581367 9503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:28.581425 9503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:28.591310 9503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:28.592644 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:28.593994 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:28.605563 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:28.605540 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:28.605599 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:28.606467 9503 server_base.cc:1034] running on GCE node
I20260926 03:38:28.607272 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:28.608742 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:28.609966 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790393908609904 us; error 39 us; skew 500 ppm
I20260926 03:38:28.610272 9503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:28 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393908, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:28.613233 9503 init.cc:377] Logged in from keytab as oryx/127.1.221.190@KRBTEST.COM (short username oryx)
I20260926 03:38:28.614622 9503 webserver.cc:460] Webserver started at http://127.1.221.190:42265/ using document root <none> and password file <none>
I20260926 03:38:28.615319 9503 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:28.615453 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:28.615743 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:28.618726 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "a54add2862974de7a432e37d51122509"
format_stamp: "Formatted at 2026-09-26 03:38:28 on dist-test-slave-lqrk"
server_key: "7e0ba08ba33e7bb71530d944e7e51f46"
server_key_iv: "264387514e2455550dbd67fb203d0393"
server_key_version: "encryptionkey@0"
I20260926 03:38:28.619599 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a54add2862974de7a432e37d51122509"
format_stamp: "Formatted at 2026-09-26 03:38:28 on dist-test-slave-lqrk"
server_key: "7e0ba08ba33e7bb71530d944e7e51f46"
server_key_iv: "264387514e2455550dbd67fb203d0393"
server_key_version: "encryptionkey@0"
I20260926 03:38:28.624891 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:38:28.628556 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:28.629981 9503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:38:28.630146 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "a54add2862974de7a432e37d51122509"
format_stamp: "Formatted at 2026-09-26 03:38:28 on dist-test-slave-lqrk"
server_key: "7e0ba08ba33e7bb71530d944e7e51f46"
server_key_iv: "264387514e2455550dbd67fb203d0393"
server_key_version: "encryptionkey@0"
I20260926 03:38:28.630357 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:28.642061 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:28.646502 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:28.646770 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:28.666941 9503 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36063
I20260926 03:38:28.666962 9579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36063 every 8 connection(s)
I20260926 03:38:28.668383 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:28.672196 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:28.678519 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9503
I20260926 03:38:28.678699 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:28.679343 1910 external_mini_cluster.cc:1428] Setting key 54218aa18914519d3f1af36ecdcf356c
I20260926 03:38:28.682760 9580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Bootstrap starting.
I20260926 03:38:28.685899 9580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:28.686966 9580 log.cc:824] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:28.689711 9580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: No bootstrap required, opened a new log
Sep 26 03:38:28 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:28.698161 9580 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } }
I20260926 03:38:28.698602 9580 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:28.698709 9580 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54add2862974de7a432e37d51122509, State: Initialized, Role: FOLLOWER
I20260926 03:38:28.699358 9580 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } }
I20260926 03:38:28.699658 9580 raft_consensus.cc:388] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:28.699785 9580 raft_consensus.cc:482] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:28.699966 9580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:28.701931 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:28.681367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.1.221.190:36063)
0926 03:38:28.682167 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:28.682180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:28.682224 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:38:28.684012 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:28.684036 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:28.684481 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:28.684845 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:28.686340 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:28.687756 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:28.688994 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:28.689681 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:28.689858 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:28.694326 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:28.694341 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:28.694362 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:28.694408 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:28.697437 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:28.698331 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:28.698338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:28.698345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:28.698550 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:28.699000 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:28.699005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:28.699009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:28.699654 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:28.699958 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:28.700434 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 03:38:28.700864 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":359,"threads_started":1}
I20260926 03:38:28.702483 9580 raft_consensus.cc:504] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } }
I20260926 03:38:28.703094 9580 leader_election.cc:302] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54add2862974de7a432e37d51122509; no voters:
I20260926 03:38:28.703749 9580 leader_election.cc:288] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:28.703899 9585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:28.705511 9585 raft_consensus.cc:686] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [term 1 LEADER]: Becoming Leader. State: Replica: a54add2862974de7a432e37d51122509, State: Running, Role: LEADER
I20260926 03:38:28.705811 9580 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:28.706117 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } }
I20260926 03:38:28.708979 9587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54add2862974de7a432e37d51122509. Latest consensus state: current_term: 1 leader_uuid: "a54add2862974de7a432e37d51122509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } } }
I20260926 03:38:28.709425 9587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:28.708976 9586 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54add2862974de7a432e37d51122509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54add2862974de7a432e37d51122509" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36063 } } }
I20260926 03:38:28.709959 9586 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:28.710614 9595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:28.715823 9595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:28.725936 9595 catalog_manager.cc:1347] Generated new cluster ID: cd8cb6ae21c74a2c9cd827e66075fd51
I20260926 03:38:28.726066 9595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:28.758755 9595 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:28.760191 9595 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:28.772385 9595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Generated new TSK 0
I20260926 03:38:28.773272 9595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
I20260926 03:38:28.840651 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36063
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:29.184834 9606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:29.185118 9606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:29.185215 9606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:29.192504 9606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:29.192615 9606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:29.192720 9606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:38:29.192826 9606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:29.202654 9606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:29.203965 9606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:29.205231 9606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:29.218076 9612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:29.218149 9611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:29.218046 9614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:29.218832 9606 server_base.cc:1034] running on GCE node
I20260926 03:38:29.219560 9606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:29.220551 9606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:29.221796 9606 hybrid_clock.cc:648] HybridClock initialized: now 1790393909221720 us; error 45 us; skew 500 ppm
I20260926 03:38:29.222136 9606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:29 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393909, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:29.225517 9606 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260926 03:38:29.226951 9606 webserver.cc:460] Webserver started at http://127.1.221.129:46757/ using document root <none> and password file <none>
I20260926 03:38:29.227715 9606 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:29.227818 9606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:29.228121 9606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:29.231062 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "e7db4070fd23441b9007250f17b523b9"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "864c936af9f1f31bf430156a4965e9db"
server_key_iv: "edc0ed3132b5111f6d5d5632a87d2878"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.231894 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "e7db4070fd23441b9007250f17b523b9"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "864c936af9f1f31bf430156a4965e9db"
server_key_iv: "edc0ed3132b5111f6d5d5632a87d2878"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.237273 9606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:38:29.241030 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:29.242559 9606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:38:29.242734 9606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "e7db4070fd23441b9007250f17b523b9"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "864c936af9f1f31bf430156a4965e9db"
server_key_iv: "edc0ed3132b5111f6d5d5632a87d2878"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.242954 9606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:29.260838 9606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:29.265233 9606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:29.265542 9606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:29.268322 9606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:29.268416 9606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:29.268548 9606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:29.268604 9606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:38:29 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393909, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:29.293473 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.269025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53567 (local address 127.1.221.190:36063)
0926 03:38:29.271608 (+ 2583us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:29.271615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:29.271641 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:38:29.273545 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:29.273551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:29.273657 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:29.273780 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:29.276687 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.277816 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.280792 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.281215 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.281324 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.286169 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:29.286178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:29.286183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:29.286228 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:29.289393 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.290500 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.290507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.290510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.290681 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.291251 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.291258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.291261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.291657 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:29.291806 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:29.292780 (+ 974us) server_negotiation.cc:299] Negotiation successful
0926 03:38:29.293050 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2456,"thread_start_us":2116,"threads_started":1}
I20260926 03:38:29.294272 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.269638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.129:53567)
0926 03:38:29.270545 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:29.270599 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:29.273012 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:29.274258 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:29.274273 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:29.275314 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:29.276459 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.276498 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.278087 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.278092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.280610 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.280626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.281473 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.281480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.281777 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.282553 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:29.282589 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:29.285856 (+ 3267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:29.289546 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.289557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.289585 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.290354 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.290789 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.290795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.290799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.291092 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.291814 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:29.291825 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:29.292296 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 03:38:29.292640 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 03:38:29.293094 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
I20260926 03:38:29.313370 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.299772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47229 (local address 127.1.221.190:36063)
0926 03:38:29.299934 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:29.299938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:29.300532 (+ 594us) server_negotiation.cc:407] Connection header received
0926 03:38:29.300673 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:29.300679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:29.300770 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:29.300883 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:29.301709 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.302893 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.304543 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.305252 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.305386 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.307228 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:29.307237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:29.307242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:29.307286 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:29.310608 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.311328 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.311333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.311336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.311513 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.311844 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.311857 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.311860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.312209 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:29.312369 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:29.312802 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 03:38:29.313014 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:38:29.313452 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.299884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.129:47229)
0926 03:38:29.300426 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:29.300441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:29.300570 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:29.300891 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:29.300894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:29.301301 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:29.301590 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.301599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.303038 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.303042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.304376 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.304393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.305429 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.305435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.305598 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.306232 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:29.306249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:29.307038 (+ 789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:29.310750 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.310755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.310758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.311219 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.311616 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.311619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.311621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.311747 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.312338 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:29.312343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:29.312558 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 03:38:29.312694 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:38:29.313023 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":283,"threads_started":1}
I20260926 03:38:29.321738 9606 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33949
I20260926 03:38:29.321759 9741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33949 every 8 connection(s)
I20260926 03:38:29.322916 9606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:29.324896 9742 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36063
I20260926 03:38:29.325090 9742 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:29.325767 9742 heartbeater.cc:511] Master 127.1.221.190:36063 requested a full tablet report, sending...
I20260926 03:38:29.326040 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9606
I20260926 03:38:29.326241 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:29.326913 1910 external_mini_cluster.cc:1428] Setting key ac66b940d3dbd931de1a3f40634fc3f1
I20260926 03:38:29.328104 9535 ts_manager.cc:194] Registered new tserver with Master: e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949)
I20260926 03:38:29.329955 9535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:53567
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
I20260926 03:38:29.372587 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36063
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:29.692449 9746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:29.692766 9746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:29.692873 9746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:29.700855 9746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:29.700975 9746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:29.701061 9746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:38:29.701143 9746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:29.711195 9746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:29.712524 9746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:29.713891 9746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:29.726944 9754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:29.726985 9751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:29.727016 9752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:29.727892 9746 server_base.cc:1034] running on GCE node
I20260926 03:38:29.728531 9746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:29.729524 9746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:29.730731 9746 hybrid_clock.cc:648] HybridClock initialized: now 1790393909730653 us; error 64 us; skew 500 ppm
I20260926 03:38:29.731036 9746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:29 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393909, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:29.734223 9746 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260926 03:38:29.735834 9746 webserver.cc:460] Webserver started at http://127.1.221.130:43511/ using document root <none> and password file <none>
I20260926 03:38:29.736510 9746 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:29.736595 9746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:29.736863 9746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:29.739876 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "69d92f4615ab4712aaca10ffdfa1695d"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "69339f15249a86035c6b76340f800390"
server_key_iv: "7cd6620c8d4c9e481af4a4e60732ecfe"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.740759 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "69d92f4615ab4712aaca10ffdfa1695d"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "69339f15249a86035c6b76340f800390"
server_key_iv: "7cd6620c8d4c9e481af4a4e60732ecfe"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.746236 9746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:29.749833 9761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:29.751196 9746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:38:29.751333 9746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "69d92f4615ab4712aaca10ffdfa1695d"
format_stamp: "Formatted at 2026-09-26 03:38:29 on dist-test-slave-lqrk"
server_key: "69339f15249a86035c6b76340f800390"
server_key_iv: "7cd6620c8d4c9e481af4a4e60732ecfe"
server_key_version: "encryptionkey@0"
I20260926 03:38:29.751601 9746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:29.777268 9746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:29.781383 9746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:29.781652 9746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:29.784461 9746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:29.784552 9746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:29.784677 9746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:29.784731 9746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:38:29 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393909, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:29.807494 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.785066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52097 (local address 127.1.221.190:36063)
0926 03:38:29.785226 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:29.785231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:29.786844 (+ 1613us) server_negotiation.cc:407] Connection header received
0926 03:38:29.788965 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:29.788972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:29.789068 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:29.789164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:29.791583 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.792542 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.794765 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.795290 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.795462 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.800675 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:29.800684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:29.800689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:29.800736 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:29.803469 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.804471 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.804477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.804480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.804651 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.805211 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.805216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.805220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.805636 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:29.805800 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:29.806967 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0926 03:38:29.807183 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:38:29.809238 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.785724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.130:52097)
0926 03:38:29.786634 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:29.786686 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:29.788658 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:29.789585 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:29.789601 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:29.790443 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:29.791363 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.791417 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.792714 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.792722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.794557 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.794572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.795573 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.795582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.795943 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.796888 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:29.796928 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:29.800363 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:29.803614 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.803628 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.803655 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.804285 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.804783 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.804788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.804795 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.805040 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.805768 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:29.805778 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:29.806404 (+ 626us) client_negotiation.cc:769] Sending connection context
0926 03:38:29.806853 (+ 449us) client_negotiation.cc:240] Negotiation successful
0926 03:38:29.807562 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":354,"threads_started":1}
I20260926 03:38:29.831540 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.815654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.130:52079)
0926 03:38:29.816271 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:29.816295 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:29.816571 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:29.817334 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:29.817339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:29.818007 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:29.818398 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.818413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.819910 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.819913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.821046 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:29.821056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.822012 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.822019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:29.822189 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.822818 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:29.822841 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:29.823864 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:29.828159 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.828165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.828170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.828716 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.829122 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:29.829127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:29.829131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:29.829383 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:29.830270 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:29.830275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:29.830493 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 03:38:29.830659 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:38:29.831033 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":347,"threads_started":1}
I20260926 03:38:29.832589 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:29.815310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52079 (local address 127.1.221.190:36063)
0926 03:38:29.815566 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:29.815573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:29.816392 (+ 819us) server_negotiation.cc:407] Connection header received
0926 03:38:29.816955 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:29.816961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:29.817049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:29.817149 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:29.818573 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.819752 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.821179 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:29.821796 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:29.821917 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:29.824064 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:29.824071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:29.824076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:29.824124 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:29.828033 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.828827 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.828833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.828837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.828992 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:29.829550 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:29.829556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:29.829559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:29.830057 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:29.830207 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:29.832054 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0926 03:38:29.832287 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:38:29.848253 9746 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36063
I20260926 03:38:29.848279 9880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36063 every 8 connection(s)
I20260926 03:38:29.849633 9746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:29.851879 9881 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36063
I20260926 03:38:29.852106 9881 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:29.852777 9881 heartbeater.cc:511] Master 127.1.221.190:36063 requested a full tablet report, sending...
I20260926 03:38:29.854148 9535 ts_manager.cc:194] Registered new tserver with Master: 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063)
I20260926 03:38:29.854866 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9746
I20260926 03:38:29.855047 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:29.855307 9535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:52097
I20260926 03:38:29.855520 1910 external_mini_cluster.cc:1428] Setting key 4319b53f0eb0ac2976415c1e25aa29ba
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260926 03:38:29.900748 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36063
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:30.315640 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:30.315932 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:30.316040 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:30.324326 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:30.324468 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:30.324604 9885 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:38:30.324731 9885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:30.333297 9742 heartbeater.cc:503] Master 127.1.221.190:36063 was elected leader, sending a full tablet report...
I20260926 03:38:30.336334 9885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:30.337589 9885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:30.338925 9885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:30.352000 9893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:30.352074 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:30.352087 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:30.352896 9885 server_base.cc:1034] running on GCE node
I20260926 03:38:30.353482 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:30.354437 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:30.355669 9885 hybrid_clock.cc:648] HybridClock initialized: now 1790393910355584 us; error 50 us; skew 500 ppm
I20260926 03:38:30.356019 9885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:30 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393910, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:30.359458 9885 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260926 03:38:30.361093 9885 webserver.cc:460] Webserver started at http://127.1.221.131:32921/ using document root <none> and password file <none>
I20260926 03:38:30.361815 9885 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:30.361924 9885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:30.362259 9885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:30.365345 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
format_stamp: "Formatted at 2026-09-26 03:38:30 on dist-test-slave-lqrk"
server_key: "f885aba705a57df77875140526944d07"
server_key_iv: "298f3c0790052e379345676541458724"
server_key_version: "encryptionkey@0"
I20260926 03:38:30.366140 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
format_stamp: "Formatted at 2026-09-26 03:38:30 on dist-test-slave-lqrk"
server_key: "f885aba705a57df77875140526944d07"
server_key_iv: "298f3c0790052e379345676541458724"
server_key_version: "encryptionkey@0"
I20260926 03:38:30.371624 9885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:38:30.375300 9900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:30.376745 9885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:30.376896 9885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
format_stamp: "Formatted at 2026-09-26 03:38:30 on dist-test-slave-lqrk"
server_key: "f885aba705a57df77875140526944d07"
server_key_iv: "298f3c0790052e379345676541458724"
server_key_version: "encryptionkey@0"
I20260926 03:38:30.377080 9885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:30.407256 9885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:30.411821 9885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:30.412091 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:30.414762 9885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:30.414877 9885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:30.415002 9885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:30.415084 9885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:38:30 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393910, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:30.441917 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:30.416297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.131:33775)
0926 03:38:30.417100 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:30.417145 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:30.419000 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:30.420004 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:30.420016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:30.420832 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:30.421742 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:30.421778 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.423133 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.423137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:30.425752 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:30.425775 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.427214 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.427224 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:30.427679 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:30.428678 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:30.428709 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:30.432118 (+ 3409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:30.436620 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:30.436630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:30.436650 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:30.437254 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:30.437700 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:30.437705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:30.437708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:30.437931 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:30.438608 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:30.438616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:30.439121 (+ 505us) client_negotiation.cc:769] Sending connection context
0926 03:38:30.439494 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 03:38:30.440122 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260926 03:38:30.443150 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:30.415469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:33775 (local address 127.1.221.190:36063)
0926 03:38:30.415876 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:30.415883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:30.417266 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 03:38:30.419451 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:30.419459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:30.419558 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:30.419696 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:30.421945 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.422910 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.426055 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.426831 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.426990 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:30.432422 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:30.432431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:30.432436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:30.432482 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:30.436062 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:30.437417 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:30.437423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:30.437426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:30.437578 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:30.438055 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:30.438060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:30.438064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:30.438406 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:30.438538 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:30.442521 (+ 3983us) server_negotiation.cc:299] Negotiation successful
0926 03:38:30.442772 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":193,"threads_started":1}
I20260926 03:38:30.466281 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:30.450627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37083 (local address 127.1.221.190:36063)
0926 03:38:30.450827 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:30.450834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:30.452019 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 03:38:30.452207 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:30.452211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:30.452288 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:30.452399 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:30.453689 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.454452 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.456542 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.457039 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.457161 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:30.459947 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:30.459955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:30.459960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:30.460005 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:30.463237 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:30.464228 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:30.464234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:30.464238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:30.464395 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:30.464847 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:30.464852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:30.464855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:30.465156 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:30.465300 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:30.465707 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 03:38:30.465938 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:38:30.466392 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:30.450984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.1.221.131:37083)
0926 03:38:30.451884 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:30.451908 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:30.452088 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:30.452435 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:30.452440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:30.453068 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:30.453482 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:30.453495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.454598 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.454604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:30.456333 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:30.456366 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.457257 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.457265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:30.457509 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:30.458554 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:30.458583 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:30.459720 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:30.463454 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:30.463460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:30.463464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:30.464040 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:30.464532 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:30.464537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:30.464541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:30.464698 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:30.465272 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:30.465276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:30.465464 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 03:38:30.465585 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 03:38:30.465962 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":548,"threads_started":1}
I20260926 03:38:30.476476 9885 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36533
I20260926 03:38:30.476506 10020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36533 every 8 connection(s)
I20260926 03:38:30.477654 9885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:30.479569 10021 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36063
I20260926 03:38:30.479774 10021 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:30.480427 10021 heartbeater.cc:511] Master 127.1.221.190:36063 requested a full tablet report, sending...
I20260926 03:38:30.481731 9535 ts_manager.cc:194] Registered new tserver with Master: f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:30.482576 9535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:33775
I20260926 03:38:30.486619 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 9885
I20260926 03:38:30.486763 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:30.487180 1910 external_mini_cluster.cc:1428] Setting key d2af818d2f8f57dd525f3e2f0cbe672d
I20260926 03:38:30.490213 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:38:30 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 03:38:30.526386 9579 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.1.221.190:36063: Invalid argument: socket address is ambiguous
Sep 26 03:38:30 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:30 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:38:30.669462 9576 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.1.221.190:36063
W20260926 03:38:30.851851 9877 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.1.221.130:36063
I20260926 03:38:30.858920 9881 heartbeater.cc:503] Master 127.1.221.190:36063 was elected leader, sending a full tablet report...
I20260926 03:38:31.484921 10021 heartbeater.cc:503] Master 127.1.221.190:36063 was elected leader, sending a full tablet report...
W20260926 03:38:31.497344 10029 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.965s user 0.005s sys 0.008s
W20260926 03:38:31.498466 9909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:30.526801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.1.221.190:36063)
0926 03:38:30.527008 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:30.527013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:30.527035 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:38:30.527164 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:30.527168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:30.527232 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:30.527362 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:30.528693 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.529434 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.530699 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:30.531128 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:30.531241 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.498136 (+966895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43920: BlockingRecv error: recv got EOF from 127.0.0.1:43920 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 03:38:31 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:31.517570 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.501839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.1.221.190:36063)
0926 03:38:31.501991 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.501996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.502383 (+ 387us) server_negotiation.cc:407] Connection header received
0926 03:38:31.502545 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.502549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.502615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.502692 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:31.503889 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.504602 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.505716 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.506139 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.506227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.512821 (+ 6594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:31.512827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:31.512831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:31.512868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:31.515322 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:31.515930 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:31.515934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:31.515936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:31.516056 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:31.516376 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:31.516381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:31.516383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:31.516668 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:31.516881 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.517084 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.517258 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:38:31.521579 9535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:31.528400 9535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:31.549826 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.541597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.1.221.131:36533)
0926 03:38:31.542685 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.542691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.542713 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:38:31.542790 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.542795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.542989 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.543848 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.544096 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.545372 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.547580 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.548748 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.548877 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.549090 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.549367 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.549538 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":827,"threads_started":1}
I20260926 03:38:31.549886 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.541363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.0.0.1:56318)
0926 03:38:31.541860 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.541878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.542005 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.543191 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.543196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.543220 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.543632 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.543644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.545499 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.545505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.547399 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.547412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.548954 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.548960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.549130 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.549178 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.549314 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.549501 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260926 03:38:31.550879 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.540334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.1.221.130:36063)
0926 03:38:31.541298 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.541307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.541347 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:38:31.541447 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.541455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.541726 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.541905 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.543255 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.544640 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.547328 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.548804 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.548930 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.549118 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.550414 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.550589 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":690,"threads_started":1}
I20260926 03:38:31.551716 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.541914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33949 (local address 127.0.0.1:43626)
0926 03:38:31.542516 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.542533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.542635 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.544930 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.544935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.544963 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.545350 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.545361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.547310 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.547315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.549265 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.549277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.551035 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.551041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.551189 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.551220 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.551283 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.551433 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":642,"threads_started":1}
I20260926 03:38:31.552232 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.540245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36063 (local address 127.0.0.1:43266)
0926 03:38:31.541025 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.541072 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.541235 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.542164 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.542171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.542201 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.543067 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.543080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.544760 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.544765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.547156 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.547168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.549264 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.549271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.549436 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.549476 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.549554 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.549684 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":3,"thread_start_us":346,"threads_started":1}
I20260926 03:38:31.552289 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.542388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.1.221.129:33949)
0926 03:38:31.544389 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.544397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.544424 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:31.544533 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.544540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.544755 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.544938 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.545542 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.547155 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.549430 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.550801 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.551654 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.551847 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.551913 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.552048 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1849,"thread_start_us":1701,"threads_started":1}
I20260926 03:38:31.553900 9951 tablet_service.cc:1467] Processing CreateTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d (DEFAULT_TABLE table=test-table [id=d88a0611437f460f998bd8c7962700bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.555089 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b5ccbcaef4f09ba93b94e1f94d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.555629 9811 tablet_service.cc:1467] Processing CreateTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d (DEFAULT_TABLE table=test-table [id=d88a0611437f460f998bd8c7962700bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.555629 9672 tablet_service.cc:1467] Processing CreateTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d (DEFAULT_TABLE table=test-table [id=d88a0611437f460f998bd8c7962700bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.556950 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b5ccbcaef4f09ba93b94e1f94d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.556972 9672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73b5ccbcaef4f09ba93b94e1f94d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.565960 10046 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Bootstrap starting.
I20260926 03:38:31.569099 10046 tablet_bootstrap.cc:654] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.570400 10046 log.cc:824] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:31.570943 10048 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Bootstrap starting.
I20260926 03:38:31.572057 10047 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Bootstrap starting.
I20260926 03:38:31.575743 10047 tablet_bootstrap.cc:654] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.576865 10047 log.cc:824] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:31.578234 10046 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: No bootstrap required, opened a new log
I20260926 03:38:31.578400 10048 tablet_bootstrap.cc:654] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.578560 10046 ts_tablet_manager.cc:1397] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260926 03:38:31.579728 10048 log.cc:824] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:31.579936 10047 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: No bootstrap required, opened a new log
I20260926 03:38:31.580195 10047 ts_tablet_manager.cc:1397] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:38:31.582623 10048 tablet_bootstrap.cc:492] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: No bootstrap required, opened a new log
I20260926 03:38:31.582875 10048 ts_tablet_manager.cc:1397] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260926 03:38:31.587235 10047 raft_consensus.cc:348] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.587643 10047 raft_consensus.cc:374] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.587777 10047 raft_consensus.cc:729] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.588366 10047 consensus_queue.cc:260] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.588313 10046 raft_consensus.cc:348] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.588752 10046 raft_consensus.cc:374] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.588857 10046 raft_consensus.cc:729] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.589442 10046 consensus_queue.cc:260] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.591738 10046 ts_tablet_manager.cc:1428] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 03:38:31.592949 10048 raft_consensus.cc:348] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.593482 10048 raft_consensus.cc:374] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.593508 10047 ts_tablet_manager.cc:1428] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 03:38:31.593638 10048 raft_consensus.cc:729] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.594257 10048 consensus_queue.cc:260] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.595952 10048 ts_tablet_manager.cc:1428] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 03:38:31.600751 10053 raft_consensus.cc:482] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:31.601022 10053 raft_consensus.cc:504] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
W20260926 03:38:31.602468 9882 tablet.cc:2369] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:38:31.602680 10053 leader_election.cc:288] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063), e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949)
I20260926 03:38:31.609416 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.603278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:40585 (local address 127.1.221.130:36063)
0926 03:38:31.603484 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.603489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.603558 (+ 69us) server_negotiation.cc:407] Connection header received
0926 03:38:31.603764 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.603770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.603842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.603942 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.604563 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.605475 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.607606 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.608593 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.608744 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.608821 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.609067 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.609189 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:38:31.609514 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.602897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36063 (local address 127.1.221.131:40585)
0926 03:38:31.603486 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.603504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.603625 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.603980 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.603986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.604033 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.604421 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.604433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.605600 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.605606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.607449 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.607466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.608773 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.608779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.608927 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.608951 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.609032 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.609163 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":272,"threads_started":1}
I20260926 03:38:31.609972 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.603601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33949 (local address 127.1.221.131:56195)
0926 03:38:31.604313 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.604327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.604430 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.604702 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.604706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.604729 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.605054 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.605066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.606298 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.606302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.607900 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.607915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.609270 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.609276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.609408 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.609430 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.609514 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.609646 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":422,"threads_started":1}
I20260926 03:38:31.610023 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.603697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:56195 (local address 127.1.221.129:33949)
0926 03:38:31.603883 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.603889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.604415 (+ 526us) server_negotiation.cc:407] Connection header received
0926 03:38:31.604535 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.604541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.604611 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.604674 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.605207 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.606180 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.608059 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.609050 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.609196 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.609266 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.609674 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.609807 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:38:31.610446 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b5ccbcaef4f09ba93b94e1f94d45d" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d" is_pre_election: true
I20260926 03:38:31.611078 9831 raft_consensus.cc:2454] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:31.611006 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b5ccbcaef4f09ba93b94e1f94d45d" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9" is_pre_election: true
I20260926 03:38:31.611574 9692 raft_consensus.cc:2454] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:31.611739 9903 leader_election.cc:302] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:31.612125 10053 raft_consensus.cc:2781] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:31.612246 10053 raft_consensus.cc:482] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:31.612341 10053 raft_consensus.cc:3037] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.614444 10053 raft_consensus.cc:504] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.615000 10053 leader_election.cc:288] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Requested vote from peers 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063), e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949)
I20260926 03:38:31.615521 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b5ccbcaef4f09ba93b94e1f94d45d" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d"
I20260926 03:38:31.615581 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73b5ccbcaef4f09ba93b94e1f94d45d" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9"
I20260926 03:38:31.615696 9831 raft_consensus.cc:3037] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.615759 9692 raft_consensus.cc:3037] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.617816 9831 raft_consensus.cc:2454] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:31.618043 9692 raft_consensus.cc:2454] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:31.618161 9903 leader_election.cc:302] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:31.618484 10053 raft_consensus.cc:2781] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:31.618950 10053 raft_consensus.cc:686] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 LEADER]: Becoming Leader. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Running, Role: LEADER
I20260926 03:38:31.619534 10053 consensus_queue.cc:237] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } }
I20260926 03:38:31.624073 9534 catalog_manager.cc:5495] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } health_report { overall_health: UNKNOWN } } }
I20260926 03:38:31.650563 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.645529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.1.221.131:36533)
0926 03:38:31.645728 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.645734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.645753 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:31.645885 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.645889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.645964 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.646110 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:31.646484 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.647223 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.648543 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.648932 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.649040 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.649488 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:31.650006 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:31.650143 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.650222 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.650304 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:38:31.658247 9831 raft_consensus.cc:1261] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:31.658247 9692 raft_consensus.cc:1261] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:31.659152 10057 consensus_queue.cc:1035] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:31.659508 10053 consensus_queue.cc:1035] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:31.670192 10060 mvcc.cc:205] Tried to move back new op lower bound from 7333453462143492096 to 7333453461997314048. Current Snapshot: MvccSnapshot[applied={T|T < 7333453462143492096}]
I20260926 03:38:31.689584 9534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:31.689859 9534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:31.693480 9534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Sending DeleteTablet for 3 replicas of tablet d73b5ccbcaef4f09ba93b94e1f94d45d
I20260926 03:38:31.694679 9811 tablet_service.cc:1514] Processing DeleteTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:31 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43266
I20260926 03:38:31.695148 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:31 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56318
I20260926 03:38:31.695654 9672 tablet_service.cc:1514] Processing DeleteTablet for tablet d73b5ccbcaef4f09ba93b94e1f94d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:31 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43626
I20260926 03:38:31.695909 9534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:38:31.696127 10069 tablet_replica.cc:330] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: stopping tablet replica
W20260926 03:38:31.696565 9534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:31.696554 10069 raft_consensus.cc:2229] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:31.697156 10069 raft_consensus.cc:2258] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:31.697937 10071 tablet_replica.cc:330] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: stopping tablet replica
I20260926 03:38:31.698380 10071 raft_consensus.cc:2229] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:31.699254 10071 raft_consensus.cc:2258] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:31.701612 10071 ts_tablet_manager.cc:1905] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:31.705276 9671 tablet_service.cc:1467] Processing CreateTablet for tablet 312c3f1df43943e485f802ebff109f69 (DEFAULT_TABLE table=test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.705921 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312c3f1df43943e485f802ebff109f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.708032 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 312c3f1df43943e485f802ebff109f69 (DEFAULT_TABLE table=test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.708484 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312c3f1df43943e485f802ebff109f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.708878 10071 ts_tablet_manager.cc:1918] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:31.709069 10071 log.cc:1196] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/d73b5ccbcaef4f09ba93b94e1f94d45d
I20260926 03:38:31.709233 10069 ts_tablet_manager.cc:1905] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:31.709645 10071 ts_tablet_manager.cc:1939] T d73b5ccbcaef4f09ba93b94e1f94d45d P f94b053c1f5446a58cf2f328c0ab80e2: Deleting consensus metadata
I20260926 03:38:31.711170 9811 tablet_service.cc:1467] Processing CreateTablet for tablet 312c3f1df43943e485f802ebff109f69 (DEFAULT_TABLE table=test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:31.711669 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 312c3f1df43943e485f802ebff109f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.711998 10070 tablet_replica.cc:330] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: stopping tablet replica
I20260926 03:38:31.712484 10070 raft_consensus.cc:2229] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:31.713030 10070 raft_consensus.cc:2258] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:31.713811 9521 catalog_manager.cc:4841] TS f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533): tablet d73b5ccbcaef4f09ba93b94e1f94d45d (table test-table [id=d88a0611437f460f998bd8c7962700bb]) successfully deleted
I20260926 03:38:31.716236 10069 ts_tablet_manager.cc:1918] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:31.716408 10069 log.cc:1196] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/d73b5ccbcaef4f09ba93b94e1f94d45d
I20260926 03:38:31.716991 10069 ts_tablet_manager.cc:1939] T d73b5ccbcaef4f09ba93b94e1f94d45d P 69d92f4615ab4712aaca10ffdfa1695d: Deleting consensus metadata
I20260926 03:38:31.718437 10048 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Bootstrap starting.
I20260926 03:38:31.718775 9521 catalog_manager.cc:4841] TS 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063): tablet d73b5ccbcaef4f09ba93b94e1f94d45d (table test-table [id=d88a0611437f460f998bd8c7962700bb]) successfully deleted
I20260926 03:38:31.719417 10046 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Bootstrap starting.
I20260926 03:38:31.719985 10048 tablet_bootstrap.cc:654] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.720813 10046 tablet_bootstrap.cc:654] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.721447 10070 ts_tablet_manager.cc:1905] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:31.724118 10047 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Bootstrap starting.
I20260926 03:38:31.725608 10047 tablet_bootstrap.cc:654] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.726747 10046 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: No bootstrap required, opened a new log
I20260926 03:38:31.726897 10046 ts_tablet_manager.cc:1397] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:38:31.727743 10046 raft_consensus.cc:348] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.727965 10046 raft_consensus.cc:374] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.728051 10046 raft_consensus.cc:729] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.728832 10070 ts_tablet_manager.cc:1918] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:31.728995 10070 log.cc:1196] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/d73b5ccbcaef4f09ba93b94e1f94d45d
I20260926 03:38:31.729552 10047 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: No bootstrap required, opened a new log
I20260926 03:38:31.729591 10070 ts_tablet_manager.cc:1939] T d73b5ccbcaef4f09ba93b94e1f94d45d P e7db4070fd23441b9007250f17b523b9: Deleting consensus metadata
I20260926 03:38:31.729693 10047 ts_tablet_manager.cc:1397] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 03:38:31.730639 10047 raft_consensus.cc:348] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.730930 10047 raft_consensus.cc:374] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.731076 10047 raft_consensus.cc:729] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.731258 10048 tablet_bootstrap.cc:492] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: No bootstrap required, opened a new log
I20260926 03:38:31.731452 10048 ts_tablet_manager.cc:1397] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260926 03:38:31.731427 9521 catalog_manager.cc:4841] TS e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949): tablet d73b5ccbcaef4f09ba93b94e1f94d45d (table test-table [id=d88a0611437f460f998bd8c7962700bb]) successfully deleted
I20260926 03:38:31.731473 10047 consensus_queue.cc:260] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.731850 10046 consensus_queue.cc:260] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.732219 10047 ts_tablet_manager.cc:1428] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:38:31.732973 10048 raft_consensus.cc:348] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.733120 10046 ts_tablet_manager.cc:1428] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 03:38:31.733189 10048 raft_consensus.cc:374] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.733268 10048 raft_consensus.cc:729] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.733563 10048 consensus_queue.cc:260] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.735018 10048 ts_tablet_manager.cc:1428] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260926 03:38:31.825991 9743 tablet.cc:2369] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:38:31.848125 10054 raft_consensus.cc:482] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:31.848357 10054 raft_consensus.cc:504] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.850323 10054 leader_election.cc:288] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:31.859709 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.851336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35393 (local address 127.1.221.129:33949)
0926 03:38:31.851522 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.851528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.851767 (+ 239us) server_negotiation.cc:407] Connection header received
0926 03:38:31.852050 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.852056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.852131 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.852188 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.852958 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.854130 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.857205 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.858419 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.858558 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.858633 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.859247 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.859404 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:38:31.860651 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.850910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33949 (local address 127.1.221.130:35393)
0926 03:38:31.851665 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.851686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.851892 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.852384 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.852389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.852417 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.852782 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.852794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.854522 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.854527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.857029 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.857045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.858846 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.858852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.859063 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.859096 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.860148 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.860300 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":621,"threads_started":1}
I20260926 03:38:31.864272 9691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312c3f1df43943e485f802ebff109f69" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9" is_pre_election: true
I20260926 03:38:31.864584 9691 raft_consensus.cc:2454] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69d92f4615ab4712aaca10ffdfa1695d in term 0.
I20260926 03:38:31.865146 9764 leader_election.cc:302] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, e7db4070fd23441b9007250f17b523b9; no voters:
I20260926 03:38:31.865527 10054 raft_consensus.cc:2781] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:31.865638 10054 raft_consensus.cc:482] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:31.865726 10054 raft_consensus.cc:3037] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.868609 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.852313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:50089 (local address 127.1.221.131:36533)
0926 03:38:31.852466 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.852471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.861170 (+ 8699us) server_negotiation.cc:407] Connection header received
0926 03:38:31.861427 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.861433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.861530 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.861590 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:31.862129 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.863224 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.866181 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.867421 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.867576 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.867655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.868146 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.868289 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:38:31.868899 10054 raft_consensus.cc:504] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.869287 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.851948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.1.221.130:50089)
0926 03:38:31.861077 (+ 9129us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.861095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.861268 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.861792 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.861797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.861822 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:31.861973 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.861983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.863454 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.863459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.865924 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.865963 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.867744 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.867751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.867966 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.867999 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.868843 (+ 844us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.869008 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9031,"thread_start_us":11439,"threads_started":1}
I20260926 03:38:31.869637 10054 leader_election.cc:288] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 election: Requested vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:31.870153 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312c3f1df43943e485f802ebff109f69" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" is_pre_election: true
I20260926 03:38:31.870337 9691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312c3f1df43943e485f802ebff109f69" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9"
I20260926 03:38:31.870533 9691 raft_consensus.cc:3037] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.870635 9976 raft_consensus.cc:2454] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69d92f4615ab4712aaca10ffdfa1695d in term 0.
I20260926 03:38:31.871567 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312c3f1df43943e485f802ebff109f69" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
I20260926 03:38:31.871765 9976 raft_consensus.cc:3037] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:31.873502 9691 raft_consensus.cc:2454] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d92f4615ab4712aaca10ffdfa1695d in term 1.
I20260926 03:38:31.874502 9976 raft_consensus.cc:2454] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d92f4615ab4712aaca10ffdfa1695d in term 1.
I20260926 03:38:31.874888 9764 leader_election.cc:302] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, e7db4070fd23441b9007250f17b523b9; no voters:
I20260926 03:38:31.875149 10054 raft_consensus.cc:2781] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:31.875852 10054 raft_consensus.cc:686] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 LEADER]: Becoming Leader. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Running, Role: LEADER
I20260926 03:38:31.876305 10054 consensus_queue.cc:237] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.879736 9532 catalog_manager.cc:5495] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d reported cstate change: term changed from 0 to 1, leader changed from <none> to 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130). New cstate: current_term: 1 leader_uuid: "69d92f4615ab4712aaca10ffdfa1695d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:38:31 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393908, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:31.960379 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.945811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.1.221.190:36063)
0926 03:38:31.946086 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:31.946093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:31.946585 (+ 492us) server_negotiation.cc:407] Connection header received
0926 03:38:31.946739 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:31.946745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:31.946855 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:31.946975 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:31.948301 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.949355 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.950640 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.951246 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.951407 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.954897 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:31.954903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:31.954906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:31.954938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:31.957562 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:31.958461 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:31.958464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:31.958467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:31.958582 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:31.959011 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:31.959015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:31.959016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:31.959269 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:31.959371 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:31.959932 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 03:38:31.960115 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:38:31.960752 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:31.945815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36063 (local address 127.0.0.1:55110)
0926 03:38:31.946469 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:31.946489 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:31.946622 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:31.947048 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:31.947052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:31.947883 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:31.948151 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.948160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.949503 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.949508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.950498 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:31.950510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:31.951498 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:31.951503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:31.951621 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:31.952291 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:31.952307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:31.954699 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:31.957703 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:31.957719 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:31.957727 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:31.958326 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:31.958689 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:31.958694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:31.958697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:31.958900 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:31.959427 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:31.959436 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:31.959657 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 03:38:31.959853 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 03:38:31.960298 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":106,"thread_start_us":382,"threads_started":1}
I20260926 03:38:31.965829 9532 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55110:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:38:31.967134 9532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:31.974076 9671 tablet_service.cc:1467] Processing CreateTablet for tablet 2d637e9e829146378117c2ada7c009ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8041603a886429f9a0450801749284f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:38:31.974545 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d637e9e829146378117c2ada7c009ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 03:38:31.980083 10022 tablet.cc:2369] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:38:31.979952 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 2d637e9e829146378117c2ada7c009ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8041603a886429f9a0450801749284f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:38:31.980149 9811 tablet_service.cc:1467] Processing CreateTablet for tablet 2d637e9e829146378117c2ada7c009ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8041603a886429f9a0450801749284f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:38:31.980568 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d637e9e829146378117c2ada7c009ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.980738 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d637e9e829146378117c2ada7c009ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:31.985896 10046 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: Bootstrap starting.
I20260926 03:38:31.987120 10047 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: Bootstrap starting.
I20260926 03:38:31.987753 10046 tablet_bootstrap.cc:654] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.987790 10048 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: Bootstrap starting.
I20260926 03:38:31.988711 10047 tablet_bootstrap.cc:654] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.989393 10048 tablet_bootstrap.cc:654] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:31.998224 10047 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: No bootstrap required, opened a new log
I20260926 03:38:31.998409 10046 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: No bootstrap required, opened a new log
I20260926 03:38:31.998404 10048 tablet_bootstrap.cc:492] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: No bootstrap required, opened a new log
I20260926 03:38:31.998587 10048 ts_tablet_manager.cc:1397] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 03:38:31.998593 10046 ts_tablet_manager.cc:1397] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20260926 03:38:31.998575 10047 ts_tablet_manager.cc:1397] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 03:38:31.999414 10046 raft_consensus.cc:348] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.999408 10048 raft_consensus.cc:348] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:31.999667 10048 raft_consensus.cc:374] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.999673 10046 raft_consensus.cc:374] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:31.999747 10048 raft_consensus.cc:729] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Initialized, Role: FOLLOWER
I20260926 03:38:31.999760 10046 raft_consensus.cc:729] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.000036 10046 consensus_queue.cc:260] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.000020 10048 consensus_queue.cc:260] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.000038 10047 raft_consensus.cc:348] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.000480 10047 raft_consensus.cc:374] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.000447 10054 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.000473 10053 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.000605 10047 raft_consensus.cc:729] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.000728 10053 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.000737 10054 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.001034 10047 consensus_queue.cc:260] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.001529 10052 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.001839 10046 ts_tablet_manager.cc:1428] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:38:32.001888 10052 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.002858 10047 ts_tablet_manager.cc:1428] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:38:32.003764 10048 ts_tablet_manager.cc:1428] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 03:38:32.122468 10053 raft_consensus.cc:482] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:32.122654 10053 raft_consensus.cc:504] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.123292 10053 leader_election.cc:288] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063)
I20260926 03:38:32.123881 9691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d637e9e829146378117c2ada7c009ad" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9" is_pre_election: true
I20260926 03:38:32.123960 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d637e9e829146378117c2ada7c009ad" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d" is_pre_election: true
I20260926 03:38:32.124132 9691 raft_consensus.cc:2454] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:32.124183 9831 raft_consensus.cc:2454] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:32.124517 9903 leader_election.cc:302] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7db4070fd23441b9007250f17b523b9, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:32.124745 10053 raft_consensus.cc:2781] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:32.124863 10053 raft_consensus.cc:482] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:32.124957 10053 raft_consensus.cc:3037] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.126868 10053 raft_consensus.cc:504] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.127345 10053 leader_election.cc:288] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Requested vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063)
I20260926 03:38:32.127892 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d637e9e829146378117c2ada7c009ad" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d"
I20260926 03:38:32.127854 9691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d637e9e829146378117c2ada7c009ad" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9"
I20260926 03:38:32.128091 9831 raft_consensus.cc:3037] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.128108 9691 raft_consensus.cc:3037] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.130838 9691 raft_consensus.cc:2454] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:32.130852 9831 raft_consensus.cc:2454] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:32.131199 9903 leader_election.cc:302] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:32.131485 10053 raft_consensus.cc:2781] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:32.131626 10053 raft_consensus.cc:686] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 LEADER]: Becoming Leader. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Running, Role: LEADER
I20260926 03:38:32.131867 10053 consensus_queue.cc:237] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.133230 10057 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: TxnStatusTablet state changed. Reason: New leader f94b053c1f5446a58cf2f328c0ab80e2. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.133453 10057 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:38:32.134579 9534 catalog_manager.cc:5495] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } health_report { overall_health: HEALTHY } } }
I20260926 03:38:32.135751 10089 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:38:32.135933 10089 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:38:32.155467 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.150072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.1.221.131:36533)
0926 03:38:32.150292 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.150296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.150316 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:38:32.150593 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.150598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.150686 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.150779 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:32.151189 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.151987 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.153836 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.154241 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.154323 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.154801 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.154920 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.154988 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.155130 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.155193 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 03:38:32.155608 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.150001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.0.0.1:56336)
0926 03:38:32.150215 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:32.150236 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:32.150373 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:32.150817 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:32.150822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:32.150836 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:38:32.151032 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.151042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.152108 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.152114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.153704 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.153719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.154370 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.154377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.154568 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.154643 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.155012 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.155031 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 03:38:32.155119 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:38:32.155273 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:38:32.159137 9691 raft_consensus.cc:1261] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.159210 9831 raft_consensus.cc:1261] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.159775 10053 consensus_queue.cc:1035] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.160060 10057 consensus_queue.cc:1035] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.162701 10052 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: TxnStatusTablet state changed. Reason: New leader f94b053c1f5446a58cf2f328c0ab80e2. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.163015 10052 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.164454 10053 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.164731 10053 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:38:32.165179 10062 mvcc.cc:205] Tried to move back new op lower bound from 7333453464198619136 to 7333453464094212096. Current Snapshot: MvccSnapshot[applied={T|T < 7333453464198619136}]
I20260926 03:38:32.165465 10052 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.165679 10052 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P e7db4070fd23441b9007250f17b523b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.170833 10054 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: TxnStatusTablet state changed. Reason: New leader f94b053c1f5446a58cf2f328c0ab80e2. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.171116 10054 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.173048 10053 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.173274 10053 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P f94b053c1f5446a58cf2f328c0ab80e2: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:38:32.178591 10054 tablet_replica.cc:439] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } } }
I20260926 03:38:32.178897 10054 tablet_replica.cc:442] T 2d637e9e829146378117c2ada7c009ad P 69d92f4615ab4712aaca10ffdfa1695d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:38:32.191792 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.184762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.1.221.130:36063)
0926 03:38:32.185875 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.185881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.185905 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:38:32.186010 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.186015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.186113 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.186287 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:32.186700 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.187900 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.189218 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.189881 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.190008 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.190382 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.191133 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.191281 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.191328 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.191449 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":845,"threads_started":1}
I20260926 03:38:32.208204 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.202589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.1.221.130:49823)
0926 03:38:32.203153 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:32.203168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:32.203301 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:32.203627 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:32.203631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:32.203642 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:38:32.203931 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.203940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.204927 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.204932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.206092 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.206105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.206883 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.206889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.207063 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.207139 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.207671 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.207682 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:38:32.207745 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:38:32.207871 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":340,"threads_started":1}
I20260926 03:38:32.208196 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.202714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49823 (local address 127.1.221.131:36533)
0926 03:38:32.202913 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.202920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.203244 (+ 324us) server_negotiation.cc:407] Connection header received
0926 03:38:32.203446 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.203460 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.203528 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.203600 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:32.204074 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.204810 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.206223 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.206722 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.206836 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.207345 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.207576 (+ 231us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.207670 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.207789 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.207882 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:38:32.219681 9691 raft_consensus.cc:1261] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Refusing update from remote peer 69d92f4615ab4712aaca10ffdfa1695d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.219816 9976 raft_consensus.cc:1261] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Refusing update from remote peer 69d92f4615ab4712aaca10ffdfa1695d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.220472 10077 consensus_queue.cc:1035] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.220818 10054 consensus_queue.cc:1035] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.265221 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.257521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36063 (local address 127.1.221.131:58469)
0926 03:38:32.258236 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:32.258256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:32.258470 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:32.258861 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:32.258866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:32.258879 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:38:32.259301 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.259313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.260812 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.260815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.262265 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.262283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.263491 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.263498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.263712 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.263807 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.264432 (+ 625us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.264446 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 03:38:32.264575 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 03:38:32.264730 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":339,"threads_started":1}
I20260926 03:38:32.265336 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.257501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:58469 (local address 127.1.221.130:36063)
0926 03:38:32.257709 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.257716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.258338 (+ 622us) server_negotiation.cc:407] Connection header received
0926 03:38:32.258619 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.258626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.258720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.258810 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:32.259573 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.260675 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.262462 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.263139 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.263250 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.264057 (+ 807us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.264210 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.264283 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.264653 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.264821 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:38:32.307343 9533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:32.307641 9533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:32.311506 9533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Sending DeleteTablet for 3 replicas of tablet 312c3f1df43943e485f802ebff109f69
I20260926 03:38:32.312305 9671 tablet_service.cc:1514] Processing DeleteTablet for tablet 312c3f1df43943e485f802ebff109f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43626
I20260926 03:38:32.312758 9811 tablet_service.cc:1514] Processing DeleteTablet for tablet 312c3f1df43943e485f802ebff109f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43266
I20260926 03:38:32.313023 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 312c3f1df43943e485f802ebff109f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56318
I20260926 03:38:32.326126 10109 tablet_replica.cc:330] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: stopping tablet replica
I20260926 03:38:32.326355 10109 raft_consensus.cc:2229] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:32.326623 10109 raft_consensus.cc:2258] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.327870 10109 ts_tablet_manager.cc:1905] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:32.335440 10109 ts_tablet_manager.cc:1918] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:32.335609 10109 log.cc:1196] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/312c3f1df43943e485f802ebff109f69
I20260926 03:38:32.336067 10109 ts_tablet_manager.cc:1939] T 312c3f1df43943e485f802ebff109f69 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting consensus metadata
I20260926 03:38:32.337262 9521 catalog_manager.cc:4841] TS f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533): tablet 312c3f1df43943e485f802ebff109f69 (table test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]) successfully deleted
I20260926 03:38:32.339326 10108 tablet_replica.cc:330] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: stopping tablet replica
I20260926 03:38:32.339583 10108 raft_consensus.cc:2229] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:32.340198 10108 raft_consensus.cc:2258] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.342074 10107 tablet_replica.cc:330] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: stopping tablet replica
I20260926 03:38:32.342253 10107 raft_consensus.cc:2229] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:32.342504 10107 raft_consensus.cc:2258] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.343432 10108 ts_tablet_manager.cc:1905] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:32.346804 10107 ts_tablet_manager.cc:1905] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:32.358547 10107 ts_tablet_manager.cc:1918] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:32.358702 10107 log.cc:1196] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/312c3f1df43943e485f802ebff109f69
I20260926 03:38:32.359131 10107 ts_tablet_manager.cc:1939] T 312c3f1df43943e485f802ebff109f69 P e7db4070fd23441b9007250f17b523b9: Deleting consensus metadata
I20260926 03:38:32.360265 9521 catalog_manager.cc:4841] TS e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949): tablet 312c3f1df43943e485f802ebff109f69 (table test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]) successfully deleted
I20260926 03:38:32.362609 10108 ts_tablet_manager.cc:1918] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:32.362746 10108 log.cc:1196] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/312c3f1df43943e485f802ebff109f69
I20260926 03:38:32.363191 10108 ts_tablet_manager.cc:1939] T 312c3f1df43943e485f802ebff109f69 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting consensus metadata
I20260926 03:38:32.364339 9521 catalog_manager.cc:4841] TS 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063): tablet 312c3f1df43943e485f802ebff109f69 (table test-table [id=e09a4fbb9ae249daadf411ce47c2f32d]) successfully deleted
Sep 26 03:38:32 dist-test-slave-lqrk krb5kdc[9490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:38:32 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:32 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:38:32.405283 9909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:32.391792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.1.221.190:36063)
0926 03:38:32.391981 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.391987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.392538 (+ 551us) server_negotiation.cc:407] Connection header received
0926 03:38:32.392738 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.392744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.392842 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.392945 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:32.394572 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.395702 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.397651 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.398266 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.398396 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.405020 (+ 6624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55114: BlockingRecv error: recv got EOF from 127.0.0.1:55114 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:38:32 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260926 03:38:32.429309 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.409714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.1.221.190:36063)
0926 03:38:32.409877 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.409883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.411841 (+ 1958us) server_negotiation.cc:407] Connection header received
0926 03:38:32.412030 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.412037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.412122 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.412229 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:32.413655 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.414753 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.416565 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.417224 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.417350 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.422207 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:32.422214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:32.422219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:32.422266 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:32.425885 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:32.426799 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:32.426805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:32.426809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:32.426975 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:32.427470 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:32.427476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:32.427480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:32.427845 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:32.428097 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.428677 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.428920 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:38:32.434964 9533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:32.435729 9533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:32.443297 9671 tablet_service.cc:1467] Processing CreateTablet for tablet 124c97aabb41455c92d03b67a6d91b1b (DEFAULT_TABLE table=test-table [id=bf3f5b322a974af699857507b4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.443800 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c97aabb41455c92d03b67a6d91b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.443755 9811 tablet_service.cc:1467] Processing CreateTablet for tablet 124c97aabb41455c92d03b67a6d91b1b (DEFAULT_TABLE table=test-table [id=bf3f5b322a974af699857507b4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.444147 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c97aabb41455c92d03b67a6d91b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.447803 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 124c97aabb41455c92d03b67a6d91b1b (DEFAULT_TABLE table=test-table [id=bf3f5b322a974af699857507b4b4f8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.448236 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124c97aabb41455c92d03b67a6d91b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.450901 10047 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Bootstrap starting.
I20260926 03:38:32.452538 10047 tablet_bootstrap.cc:654] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.458838 10048 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Bootstrap starting.
I20260926 03:38:32.460682 10048 tablet_bootstrap.cc:654] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.464038 10046 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Bootstrap starting.
I20260926 03:38:32.465878 10046 tablet_bootstrap.cc:654] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.473834 10046 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: No bootstrap required, opened a new log
I20260926 03:38:32.474028 10046 ts_tablet_manager.cc:1397] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260926 03:38:32.474915 10046 raft_consensus.cc:348] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.475234 10046 raft_consensus.cc:374] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.475332 10046 raft_consensus.cc:729] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.475610 10046 consensus_queue.cc:260] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.476358 10046 ts_tablet_manager.cc:1428] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 03:38:32.484829 10048 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: No bootstrap required, opened a new log
I20260926 03:38:32.485008 10048 ts_tablet_manager.cc:1397] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20260926 03:38:32.485838 10048 raft_consensus.cc:348] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.487201 10048 raft_consensus.cc:374] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.487315 10048 raft_consensus.cc:729] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.487617 10048 consensus_queue.cc:260] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.491436 10048 ts_tablet_manager.cc:1428] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 03:38:32.496312 10047 tablet_bootstrap.cc:492] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: No bootstrap required, opened a new log
I20260926 03:38:32.496500 10047 ts_tablet_manager.cc:1397] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.030s
I20260926 03:38:32.497345 10047 raft_consensus.cc:348] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.497635 10047 raft_consensus.cc:374] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.497727 10047 raft_consensus.cc:729] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.498183 10047 consensus_queue.cc:260] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.498798 10047 ts_tablet_manager.cc:1428] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:38:32.669567 10057 raft_consensus.cc:482] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:32.669790 10057 raft_consensus.cc:504] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.670547 10057 leader_election.cc:288] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063)
I20260926 03:38:32.671128 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c97aabb41455c92d03b67a6d91b1b" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9" is_pre_election: true
I20260926 03:38:32.671273 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c97aabb41455c92d03b67a6d91b1b" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d" is_pre_election: true
I20260926 03:38:32.671368 9692 raft_consensus.cc:2454] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:32.671574 9831 raft_consensus.cc:2454] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 0.
I20260926 03:38:32.671996 9903 leader_election.cc:302] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7db4070fd23441b9007250f17b523b9, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:32.672293 10057 raft_consensus.cc:2781] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:32.672411 10057 raft_consensus.cc:482] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:32.672505 10057 raft_consensus.cc:3037] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.675599 10057 raft_consensus.cc:504] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.676285 10057 leader_election.cc:288] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Requested vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063)
I20260926 03:38:32.676822 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c97aabb41455c92d03b67a6d91b1b" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9"
I20260926 03:38:32.676836 9831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124c97aabb41455c92d03b67a6d91b1b" candidate_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d"
I20260926 03:38:32.677021 9692 raft_consensus.cc:3037] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.677049 9831 raft_consensus.cc:3037] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.680004 9692 raft_consensus.cc:2454] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:32.680244 9831 raft_consensus.cc:2454] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94b053c1f5446a58cf2f328c0ab80e2 in term 1.
I20260926 03:38:32.680517 9903 leader_election.cc:302] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7db4070fd23441b9007250f17b523b9, f94b053c1f5446a58cf2f328c0ab80e2; no voters:
I20260926 03:38:32.680814 10057 raft_consensus.cc:2781] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:32.681061 10057 raft_consensus.cc:686] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 LEADER]: Becoming Leader. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Running, Role: LEADER
I20260926 03:38:32.681375 10057 consensus_queue.cc:237] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.684341 9532 catalog_manager.cc:5495] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } health_report { overall_health: HEALTHY } } }
I20260926 03:38:32.731937 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.726378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.1.221.131:36533)
0926 03:38:32.726844 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.726848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.726864 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:38:32.726919 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.726923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.726983 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.727062 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:32.727526 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.728219 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.730126 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.730519 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.730626 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.731150 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:32.731272 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:32.731352 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.731569 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.731644 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":270,"threads_started":1}
I20260926 03:38:32.734583 9831 raft_consensus.cc:1261] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.734939 9692 raft_consensus.cc:1261] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Refusing update from remote peer f94b053c1f5446a58cf2f328c0ab80e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:32.735409 10057 consensus_queue.cc:1035] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.735685 10053 consensus_queue.cc:1035] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:32.741658 10061 mvcc.cc:205] Tried to move back new op lower bound from 7333453466555461632 to 7333453466346065920. Current Snapshot: MvccSnapshot[applied={T|T < 7333453466555461632}]
I20260926 03:38:32.748991 9532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:32.749221 9532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:32.755352 9671 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c97aabb41455c92d03b67a6d91b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43626
I20260926 03:38:32.755434 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c97aabb41455c92d03b67a6d91b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56318
I20260926 03:38:32.755676 10107 tablet_replica.cc:330] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: stopping tablet replica
I20260926 03:38:32.755872 10109 tablet_replica.cc:330] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: stopping tablet replica
I20260926 03:38:32.755864 10107 raft_consensus.cc:2229] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:32.756062 10107 raft_consensus.cc:2258] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.756062 10109 raft_consensus.cc:2229] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:32.756350 10109 raft_consensus.cc:2258] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.758193 10109 ts_tablet_manager.cc:1905] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:32.754436 9532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Sending DeleteTablet for 3 replicas of tablet 124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.760825 10107 ts_tablet_manager.cc:1905] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:32.764285 9811 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c97aabb41455c92d03b67a6d91b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43266
I20260926 03:38:32.764575 10108 tablet_replica.cc:330] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: stopping tablet replica
I20260926 03:38:32.764753 10108 raft_consensus.cc:2229] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:32.764987 10108 raft_consensus.cc:2258] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:32.765254 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c97aabb41455c92d03b67a6d91b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56318
I20260926 03:38:32.765080 9671 tablet_service.cc:1514] Processing DeleteTablet for tablet 124c97aabb41455c92d03b67a6d91b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:32 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43626
I20260926 03:38:32.765901 10107 ts_tablet_manager.cc:1918] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:32.766045 10107 log.cc:1196] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.766439 10107 ts_tablet_manager.cc:1939] T 124c97aabb41455c92d03b67a6d91b1b P e7db4070fd23441b9007250f17b523b9: Deleting consensus metadata
I20260926 03:38:32.766652 10109 ts_tablet_manager.cc:1918] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:32.766863 10109 log.cc:1196] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.767556 9521 catalog_manager.cc:4841] TS e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949): tablet 124c97aabb41455c92d03b67a6d91b1b (table test-table [id=bf3f5b322a974af699857507b4b4f8f6]) successfully deleted
I20260926 03:38:32.767588 10108 ts_tablet_manager.cc:1905] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 03:38:32.768211 9521 catalog_manager.cc:4808] TS e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949): delete failed for tablet 124c97aabb41455c92d03b67a6d91b1b because the tablet was not found. No further retry: Not found: Tablet not found: 124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.768025 9532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:38:32.768760 9532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:38:32.768807 10109 ts_tablet_manager.cc:1939] T 124c97aabb41455c92d03b67a6d91b1b P f94b053c1f5446a58cf2f328c0ab80e2: Deleting consensus metadata
I20260926 03:38:32.769974 9521 catalog_manager.cc:4841] TS f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533): tablet 124c97aabb41455c92d03b67a6d91b1b (table test-table [id=bf3f5b322a974af699857507b4b4f8f6]) successfully deleted
W20260926 03:38:32.770699 9521 catalog_manager.cc:4808] TS f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533): delete failed for tablet 124c97aabb41455c92d03b67a6d91b1b because the tablet was not found. No further retry: Not found: Tablet not found: 124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.776120 9671 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 (DEFAULT_TABLE table=test-table [id=3521cd116cdc4ed081a38ee6d2341415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.776258 10108 ts_tablet_manager.cc:1918] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:38:32.776571 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb91db7df2245ccb09e148ecfbd8d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.776669 9811 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 (DEFAULT_TABLE table=test-table [id=3521cd116cdc4ed081a38ee6d2341415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.777084 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb91db7df2245ccb09e148ecfbd8d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.779806 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 (DEFAULT_TABLE table=test-table [id=3521cd116cdc4ed081a38ee6d2341415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:38:32.780213 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb91db7df2245ccb09e148ecfbd8d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:32.782079 10047 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Bootstrap starting.
I20260926 03:38:32.783574 10048 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Bootstrap starting.
I20260926 03:38:32.783665 10047 tablet_bootstrap.cc:654] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.785066 10048 tablet_bootstrap.cc:654] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.785643 10046 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Bootstrap starting.
I20260926 03:38:32.786311 10108 log.cc:1196] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/124c97aabb41455c92d03b67a6d91b1b
I20260926 03:38:32.786844 10108 ts_tablet_manager.cc:1939] T 124c97aabb41455c92d03b67a6d91b1b P 69d92f4615ab4712aaca10ffdfa1695d: Deleting consensus metadata
I20260926 03:38:32.787129 10046 tablet_bootstrap.cc:654] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:32.788074 9521 catalog_manager.cc:4841] TS 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063): tablet 124c97aabb41455c92d03b67a6d91b1b (table test-table [id=bf3f5b322a974af699857507b4b4f8f6]) successfully deleted
I20260926 03:38:32.789873 10048 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: No bootstrap required, opened a new log
I20260926 03:38:32.790032 10048 ts_tablet_manager.cc:1397] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 03:38:32.790191 10046 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: No bootstrap required, opened a new log
I20260926 03:38:32.790355 10046 ts_tablet_manager.cc:1397] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 03:38:32.790616 10047 tablet_bootstrap.cc:492] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: No bootstrap required, opened a new log
I20260926 03:38:32.790818 10047 ts_tablet_manager.cc:1397] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260926 03:38:32.790761 10048 raft_consensus.cc:348] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.791030 10048 raft_consensus.cc:374] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.791108 10048 raft_consensus.cc:729] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d92f4615ab4712aaca10ffdfa1695d, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.791087 10046 raft_consensus.cc:348] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.791337 10046 raft_consensus.cc:374] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.791433 10046 raft_consensus.cc:729] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94b053c1f5446a58cf2f328c0ab80e2, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.791361 10048 consensus_queue.cc:260] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.791687 10046 consensus_queue.cc:260] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.792071 10047 raft_consensus.cc:348] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.792361 10047 raft_consensus.cc:374] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:32.792577 10047 raft_consensus.cc:729] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Initialized, Role: FOLLOWER
I20260926 03:38:32.792851 10047 consensus_queue.cc:260] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.794051 10047 ts_tablet_manager.cc:1428] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:38:32.794123 10046 ts_tablet_manager.cc:1428] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:38:32.794901 10048 ts_tablet_manager.cc:1428] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260926 03:38:32.952445 10052 raft_consensus.cc:482] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:32.952649 10052 raft_consensus.cc:504] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.954270 10052 leader_election.cc:288] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:32.960259 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.954613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47459 (local address 127.1.221.130:36063)
0926 03:38:32.955077 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.955081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.955182 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:38:32.955412 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.955418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.955509 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.955637 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:32.956197 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.956949 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.958572 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.959329 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.959522 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.959596 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.959838 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.959973 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260926 03:38:32.960337 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.954533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36063 (local address 127.1.221.129:47459)
0926 03:38:32.955091 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:32.955111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:32.955245 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:32.955667 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:32.955672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:32.955700 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:32.956027 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.956041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.957082 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.957088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.958414 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.958425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.959523 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.959529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.959701 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.959723 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:38:32.959804 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:38:32.959940 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":18,"thread_start_us":264,"threads_started":1}
I20260926 03:38:32.960724 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.955332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:41385 (local address 127.1.221.131:36533)
0926 03:38:32.955529 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:32.955534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:32.955883 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:38:32.956077 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:32.956081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:32.956137 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:32.956205 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:38:32.956672 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.957395 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.959054 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.959840 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.959975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.960044 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:32.960355 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 03:38:32.960491 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:38:32.961107 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:32.955208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.1.221.129:41385)
0926 03:38:32.955775 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:32.955793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:32.955954 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:32.956240 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:32.956244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:32.956267 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:38:32.956540 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.956548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.957531 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.957536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.958898 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:32.958909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:32.960091 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:32.960098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:32.960205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:32.960229 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:38:32.960310 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:38:32.960736 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":21,"thread_start_us":292,"threads_started":1}
I20260926 03:38:32.961181 10077 raft_consensus.cc:482] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:38:32.961377 10077 raft_consensus.cc:504] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.961421 9830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "e7db4070fd23441b9007250f17b523b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d" is_pre_election: true
I20260926 03:38:32.961683 9830 raft_consensus.cc:2454] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7db4070fd23441b9007250f17b523b9 in term 0.
I20260926 03:38:32.961979 10077 leader_election.cc:288] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:32.962275 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "e7db4070fd23441b9007250f17b523b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" is_pre_election: true
I20260926 03:38:32.962386 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9" is_pre_election: true
I20260926 03:38:32.962409 9624 leader_election.cc:302] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, e7db4070fd23441b9007250f17b523b9; no voters:
I20260926 03:38:32.962452 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" is_pre_election: true
I20260926 03:38:32.962530 9976 raft_consensus.cc:2454] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7db4070fd23441b9007250f17b523b9 in term 0.
I20260926 03:38:32.962615 9692 raft_consensus.cc:2454] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69d92f4615ab4712aaca10ffdfa1695d in term 0.
I20260926 03:38:32.962812 10052 raft_consensus.cc:2781] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:32.962932 10052 raft_consensus.cc:482] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:32.963027 10052 raft_consensus.cc:3037] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.963115 9764 leader_election.cc:302] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d, e7db4070fd23441b9007250f17b523b9; no voters: f94b053c1f5446a58cf2f328c0ab80e2
I20260926 03:38:32.963414 10077 raft_consensus.cc:2781] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:38:32.963521 10077 raft_consensus.cc:482] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:38:32.963626 10077 raft_consensus.cc:3037] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.966010 10052 raft_consensus.cc:504] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.966209 10077 raft_consensus.cc:504] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.966612 10052 leader_election.cc:288] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [CANDIDATE]: Term 1 election: Requested vote from peers 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:32.966813 10077 leader_election.cc:288] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 election: Requested vote from peers e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949), f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533)
I20260926 03:38:32.967020 9830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "e7db4070fd23441b9007250f17b523b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d92f4615ab4712aaca10ffdfa1695d"
I20260926 03:38:32.967100 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "e7db4070fd23441b9007250f17b523b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
I20260926 03:38:32.967265 9830 raft_consensus.cc:2379] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7db4070fd23441b9007250f17b523b9 in current term 1: Already voted for candidate 69d92f4615ab4712aaca10ffdfa1695d in this term.
I20260926 03:38:32.967308 9975 raft_consensus.cc:3037] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:32.967417 9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7db4070fd23441b9007250f17b523b9"
I20260926 03:38:32.967489 9976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb91db7df2245ccb09e148ecfbd8d95" candidate_uuid: "69d92f4615ab4712aaca10ffdfa1695d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94b053c1f5446a58cf2f328c0ab80e2"
I20260926 03:38:32.967648 9692 raft_consensus.cc:2379] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69d92f4615ab4712aaca10ffdfa1695d in current term 1: Already voted for candidate e7db4070fd23441b9007250f17b523b9 in this term.
I20260926 03:38:32.968043 9764 leader_election.cc:302] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69d92f4615ab4712aaca10ffdfa1695d; no voters: e7db4070fd23441b9007250f17b523b9, f94b053c1f5446a58cf2f328c0ab80e2
I20260926 03:38:32.968282 10077 raft_consensus.cc:2726] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 03:38:32.969480 9975 raft_consensus.cc:2454] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7db4070fd23441b9007250f17b523b9 in term 1.
I20260926 03:38:32.969782 9624 leader_election.cc:302] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e7db4070fd23441b9007250f17b523b9, f94b053c1f5446a58cf2f328c0ab80e2; no voters: 69d92f4615ab4712aaca10ffdfa1695d
I20260926 03:38:32.970052 10052 raft_consensus.cc:2781] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:32.970410 10052 raft_consensus.cc:686] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 LEADER]: Becoming Leader. State: Replica: e7db4070fd23441b9007250f17b523b9, State: Running, Role: LEADER
I20260926 03:38:32.970736 10052 consensus_queue.cc:237] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } }
I20260926 03:38:32.974020 9534 catalog_manager.cc:5495] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7db4070fd23441b9007250f17b523b9 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "e7db4070fd23441b9007250f17b523b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db4070fd23441b9007250f17b523b9" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 } health_report { overall_health: UNKNOWN } } }
I20260926 03:38:33.038512 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.032988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.1.221.129:33949)
0926 03:38:33.033523 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.033527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.033542 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:38:33.033617 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.033621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.033687 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.033846 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:33.034283 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.035020 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.036351 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.036750 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.036856 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.037283 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:33.037878 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:33.038015 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.038123 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.038199 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":300,"threads_started":1}
I20260926 03:38:33.053278 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.048171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:45299 (local address 127.1.221.131:36533)
0926 03:38:33.048363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.048369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.048741 (+ 372us) server_negotiation.cc:407] Connection header received
0926 03:38:33.048934 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.048939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.049016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.049093 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:33.049585 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.050297 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.051655 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.052078 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.052187 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.052596 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:33.052701 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:33.052773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.052901 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.052991 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:38:33.053376 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.048010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36533 (local address 127.1.221.129:45299)
0926 03:38:33.048646 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:33.048663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:33.048802 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:33.049119 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:33.049123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:33.049134 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:38:33.049428 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:33.049437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.050433 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.050437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:33.051519 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:33.051531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.052248 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.052254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:33.052373 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.052435 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:38:33.052791 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:38:33.052802 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:38:33.052867 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:38:33.053000 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260926 03:38:33.061671 9975 raft_consensus.cc:1261] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Refusing update from remote peer e7db4070fd23441b9007250f17b523b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:33.061676 9830 raft_consensus.cc:1261] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Refusing update from remote peer e7db4070fd23441b9007250f17b523b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:38:33.062232 10052 consensus_queue.cc:1035] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69d92f4615ab4712aaca10ffdfa1695d" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:33.062551 10136 consensus_queue.cc:1035] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94b053c1f5446a58cf2f328c0ab80e2" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:38:33.086229 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.081307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:56943 (local address 127.1.221.129:33949)
0926 03:38:33.081469 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.081474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.081755 (+ 281us) server_negotiation.cc:407] Connection header received
0926 03:38:33.081870 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.081874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.081934 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.082026 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:38:33.082337 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.083079 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.084814 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.085311 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.085404 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.085730 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:38:33.085821 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:38:33.085879 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.085971 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.086033 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:38:33.086314 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.081167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33949 (local address 127.1.221.131:56943)
0926 03:38:33.081670 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:33.081686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:33.081789 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:33.082029 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:33.082032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:33.082041 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:38:33.082254 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:33.082262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.083204 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.083209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:33.084673 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:33.084687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.085446 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.085450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:33.085576 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.085614 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:38:33.085887 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:38:33.085895 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:38:33.085949 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 03:38:33.086050 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260926 03:38:33.113373 9532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:33.113677 9532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:38:33.116624 9532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a54add2862974de7a432e37d51122509: Sending DeleteTablet for 3 replicas of tablet 6bb91db7df2245ccb09e148ecfbd8d95
I20260926 03:38:33.117411 9671 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:33 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43626
I20260926 03:38:33.117590 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:33 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56318
I20260926 03:38:33.117717 9811 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb91db7df2245ccb09e148ecfbd8d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:38:33 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43266
I20260926 03:38:33.117882 10109 tablet_replica.cc:330] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: stopping tablet replica
I20260926 03:38:33.117995 10108 tablet_replica.cc:330] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: stopping tablet replica
I20260926 03:38:33.118009 10107 tablet_replica.cc:330] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: stopping tablet replica
I20260926 03:38:33.118057 10109 raft_consensus.cc:2229] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:33.118158 10108 raft_consensus.cc:2229] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:38:33.118165 10107 raft_consensus.cc:2229] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:38:33.118254 10109 raft_consensus.cc:2258] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:33.118342 10108 raft_consensus.cc:2258] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:33.118664 10107 raft_consensus.cc:2258] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:38:33.119478 10109 ts_tablet_manager.cc:1905] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:33.119860 10108 ts_tablet_manager.cc:1905] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:33.124387 10108 ts_tablet_manager.cc:1918] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:33.124521 10108 log.cc:1196] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/6bb91db7df2245ccb09e148ecfbd8d95
I20260926 03:38:33.124840 10108 ts_tablet_manager.cc:1939] T 6bb91db7df2245ccb09e148ecfbd8d95 P 69d92f4615ab4712aaca10ffdfa1695d: Deleting consensus metadata
I20260926 03:38:33.125797 9521 catalog_manager.cc:4841] TS 69d92f4615ab4712aaca10ffdfa1695d (127.1.221.130:36063): tablet 6bb91db7df2245ccb09e148ecfbd8d95 (table test-table [id=3521cd116cdc4ed081a38ee6d2341415]) successfully deleted
I20260926 03:38:33.125921 10109 ts_tablet_manager.cc:1918] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:33.126055 10109 log.cc:1196] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/6bb91db7df2245ccb09e148ecfbd8d95
I20260926 03:38:33.126179 10107 ts_tablet_manager.cc:1905] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:38:33.126484 10109 ts_tablet_manager.cc:1939] T 6bb91db7df2245ccb09e148ecfbd8d95 P f94b053c1f5446a58cf2f328c0ab80e2: Deleting consensus metadata
I20260926 03:38:33.127573 9521 catalog_manager.cc:4841] TS f94b053c1f5446a58cf2f328c0ab80e2 (127.1.221.131:36533): tablet 6bb91db7df2245ccb09e148ecfbd8d95 (table test-table [id=3521cd116cdc4ed081a38ee6d2341415]) successfully deleted
I20260926 03:38:33.130870 10107 ts_tablet_manager.cc:1918] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:38:33.130995 10107 log.cc:1196] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/6bb91db7df2245ccb09e148ecfbd8d95
I20260926 03:38:33.131471 10107 ts_tablet_manager.cc:1939] T 6bb91db7df2245ccb09e148ecfbd8d95 P e7db4070fd23441b9007250f17b523b9: Deleting consensus metadata
I20260926 03:38:33.132164 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9606
I20260926 03:38:33.132896 9521 catalog_manager.cc:4841] TS e7db4070fd23441b9007250f17b523b9 (127.1.221.129:33949): tablet 6bb91db7df2245ccb09e148ecfbd8d95 (table test-table [id=3521cd116cdc4ed081a38ee6d2341415]) successfully deleted
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.129@KRBTEST.COM
I20260926 03:38:33.148598 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.134302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.1.221.129:33949)
0926 03:38:33.134469 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.134475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.134493 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:33.134643 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.134649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.134754 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.134856 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:33.136007 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.136740 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.137894 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.138328 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.138438 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.142471 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:33.142484 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:33.142501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:33.142539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:33.145726 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.146444 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.146454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.146459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.146590 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.147002 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.147008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.147012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.147409 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:33.147713 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.148018 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.148240 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:38:33.149294 9737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:33.257084 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9606
I20260926 03:38:33.275543 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9746
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.130@KRBTEST.COM
I20260926 03:38:33.289271 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.276250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.1.221.130:36063)
0926 03:38:33.276485 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.276491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.276509 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:38:33.276658 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.276664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.276745 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.276830 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:33.278030 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.278715 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.280068 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.280507 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.280624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.284592 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:33.284606 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:33.284622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:33.284662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:33.287113 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.287730 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.287737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.287740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.287867 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.288112 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.288115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.288117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.288473 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:33.288713 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.288870 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.289032 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:38:33.289932 9876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:33.410444 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9746
I20260926 03:38:33.428468 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9885
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[9490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.131@KRBTEST.COM
I20260926 03:38:33.441856 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.429134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.1.221.131:36533)
0926 03:38:33.429282 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:33.429286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:33.429341 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:38:33.429485 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:33.429489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:33.429553 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:33.429635 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:33.430771 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.431508 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.432632 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:33.433003 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:33.433097 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:33.437108 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:33.437123 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:33.437140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:33.437180 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:33.439711 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.440276 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.440283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.440285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.440412 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:33.440660 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:33.440663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:33.440664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:33.441044 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:33.441255 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:33.441468 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:38:33.441628 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:38:33.442440 10016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:33.581620 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.580831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33949 (local address 127.1.221.131:47361)
0926 03:38:33.581288 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:33.581416 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":263,"threads_started":1}
I20260926 03:38:33.582083 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:33.581431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36063 (local address 127.1.221.131:42753)
0926 03:38:33.581888 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:33.581968 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.130:36063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"threads_started":1}
I20260926 03:38:33.584007 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9885
I20260926 03:38:33.603760 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 9503
I20260926 03:38:33.604949 9575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:33.718786 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 9503
2026-09-26T03:38:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6684 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[10160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[10160](info): set up 2 sockets
Sep 26 03:38:33 dist-test-slave-lqrk krb5kdc[10160](info): commencing operation
krb5kdc: starting...
W20260926 03:38:34.716254 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:34 dist-test-slave-lqrk krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:38:34.929160 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40079
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:35.241832 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:35.242130 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:35.242259 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:35.249508 10173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:35.249624 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:35.249693 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:35.249770 10173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:35.249831 10173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:35.259795 10173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:35.261119 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:35.262428 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:35.273654 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:35.273607 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:35.273681 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:35.274487 10173 server_base.cc:1034] running on GCE node
I20260926 03:38:35.275214 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:35.276643 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:35.277850 10173 hybrid_clock.cc:648] HybridClock initialized: now 1790393915277787 us; error 52 us; skew 500 ppm
I20260926 03:38:35.278190 10173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:38:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1629 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:38:35 dist-test-slave-lqrk krb5kdc[10187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:38:35 dist-test-slave-lqrk krb5kdc[10187](info): set up 2 sockets
Sep 26 03:38:35 dist-test-slave-lqrk krb5kdc[10187](info): commencing operation
krb5kdc: starting...
W20260926 03:38:36.376227 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:38:36 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:38:36.580592 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36665
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:36.889369 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:36.889650 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:36.889742 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:36.896811 10200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:36.896910 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:36.896955 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:36.897003 10200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:36.897045 10200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:36.906625 10200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:36.907909 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:36.909358 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:36.920380 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:36.920401 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:36.920476 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:36.921625 10200 server_base.cc:1034] running on GCE node
I20260926 03:38:36.922372 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:36.923738 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:36.924940 10200 hybrid_clock.cc:648] HybridClock initialized: now 1790393916924885 us; error 39 us; skew 500 ppm
I20260926 03:38:36.925254 10200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:36 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:36.933846 10200 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:38:36.935261 10200 webserver.cc:460] Webserver started at http://127.1.221.190:43947/ using document root <none> and password file <none>
I20260926 03:38:36.935956 10200 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:36.936069 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:36.936354 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:36.939258 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "911c2733835640c0b0e628b1fbc71ff2"
format_stamp: "Formatted at 2026-09-26 03:38:36 on dist-test-slave-lqrk"
server_key: "f16ea951bf1a8de3cf6265c860a1f6d4"
server_key_iv: "94e6d373e4a89f875d91ec19522bbe7d"
server_key_version: "encryptionkey@0"
I20260926 03:38:36.940071 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "911c2733835640c0b0e628b1fbc71ff2"
format_stamp: "Formatted at 2026-09-26 03:38:36 on dist-test-slave-lqrk"
server_key: "f16ea951bf1a8de3cf6265c860a1f6d4"
server_key_iv: "94e6d373e4a89f875d91ec19522bbe7d"
server_key_version: "encryptionkey@0"
I20260926 03:38:36.944883 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:38:36.948251 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:36.949563 10200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:36.949723 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "911c2733835640c0b0e628b1fbc71ff2"
format_stamp: "Formatted at 2026-09-26 03:38:36 on dist-test-slave-lqrk"
server_key: "f16ea951bf1a8de3cf6265c860a1f6d4"
server_key_iv: "94e6d373e4a89f875d91ec19522bbe7d"
server_key_version: "encryptionkey@0"
I20260926 03:38:36.949899 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:36.959975 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:36.963646 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:36.963877 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:36.980265 10200 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36665
I20260926 03:38:36.980273 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36665 every 8 connection(s)
I20260926 03:38:36.981545 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:36.985067 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:36.988135 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10200
I20260926 03:38:36.988339 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:36.989146 1910 external_mini_cluster.cc:1428] Setting key db44837b9530a7c9e5484fe24a8bdcfe
I20260926 03:38:36.997203 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Bootstrap starting.
I20260926 03:38:37.000896 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:37.002164 10267 log.cc:824] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:38:36 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:37.005340 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: No bootstrap required, opened a new log
I20260926 03:38:37.012518 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:36.991079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.1.221.190:36665)
0926 03:38:36.991804 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:36.991816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:36.991855 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:38:36.993418 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:36.993440 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:36.993828 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:36.994144 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:36.995817 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:36.997059 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:36.998741 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:36.999481 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:36.999609 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:37.004053 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:37.004063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:37.004082 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:37.004127 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:37.007937 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:37.008647 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:37.008656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:37.008665 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:37.008888 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:37.009269 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:37.009274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:37.009281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:37.009959 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:37.010252 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:37.010646 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:38:37.011101 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260926 03:38:37.015303 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:37.015714 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:37.015789 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911c2733835640c0b0e628b1fbc71ff2, State: Initialized, Role: FOLLOWER
I20260926 03:38:37.016304 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:37.016503 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:37.016594 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:37.016713 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:37.018750 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:37.019230 10267 leader_election.cc:302] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911c2733835640c0b0e628b1fbc71ff2; no voters:
I20260926 03:38:37.019795 10267 leader_election.cc:288] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:37.019922 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:37.020951 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 911c2733835640c0b0e628b1fbc71ff2, State: Running, Role: LEADER
I20260926 03:38:37.021476 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:37.021921 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:37.024418 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911c2733835640c0b0e628b1fbc71ff2. Latest consensus state: current_term: 1 leader_uuid: "911c2733835640c0b0e628b1fbc71ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } } }
I20260926 03:38:37.024420 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911c2733835640c0b0e628b1fbc71ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } } }
I20260926 03:38:37.024937 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:37.024940 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:37.027345 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:37.031673 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:37.040813 10279 catalog_manager.cc:1347] Generated new cluster ID: ef6713104c4b44339e4933f4fef4cd53
I20260926 03:38:37.040951 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:37.072832 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:37.074733 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:37.088971 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Generated new TSK 0
I20260926 03:38:37.090036 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:38:37.142494 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:37.461992 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:37.462257 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:37.462390 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:37.469594 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:37.469700 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:37.469846 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:37.479336 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:37.480638 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:37.481873 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:37.494195 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:37.494300 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:37.494303 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:37.494942 10293 server_base.cc:1034] running on GCE node
I20260926 03:38:37.495694 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:37.496714 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:37.497910 10293 hybrid_clock.cc:648] HybridClock initialized: now 1790393917497862 us; error 48 us; skew 500 ppm
I20260926 03:38:37.498214 10293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:37 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:37.501358 10293 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:38:37.502835 10293 webserver.cc:460] Webserver started at http://127.1.221.129:34041/ using document root <none> and password file <none>
I20260926 03:38:37.503583 10293 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:37.503731 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:37.504011 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:37.506891 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "3f24f75635644a87904f374b15ed6742"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "7d11d339f2c32f43f1976fd6119a4589"
server_key_iv: "7dfa46d5757d4fdc599b587a529cf010"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.507679 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "3f24f75635644a87904f374b15ed6742"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "7d11d339f2c32f43f1976fd6119a4589"
server_key_iv: "7dfa46d5757d4fdc599b587a529cf010"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.512822 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:38:37.516167 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:37.517630 10293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:37.517797 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "3f24f75635644a87904f374b15ed6742"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "7d11d339f2c32f43f1976fd6119a4589"
server_key_iv: "7dfa46d5757d4fdc599b587a529cf010"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.517992 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:37.534355 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:37.538718 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:37.538995 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:37.539970 10293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:37.541302 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:37.541383 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:37.541500 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:37.541558 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:37.571169 10293 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:44017
I20260926 03:38:37.571204 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:44017 every 8 connection(s)
I20260926 03:38:37.572422 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:37.582113 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10293
I20260926 03:38:37.582326 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:37.582950 1910 external_mini_cluster.cc:1428] Setting key 573bf913d8e90569dbbd45fc3bb06fa3
Sep 26 03:38:37 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:37.591207 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:37.574762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38043 (local address 127.1.221.190:36665)
0926 03:38:37.575182 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:37.575187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:37.576169 (+ 982us) server_negotiation.cc:407] Connection header received
0926 03:38:37.577863 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:37.577867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:37.577930 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:37.578022 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:37.579430 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:37.580135 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:37.581587 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:37.582153 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:37.582278 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:37.585729 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:37.585734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:37.585737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:37.585776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:37.588324 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:37.588981 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:37.588986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:37.588990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:37.589103 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:37.589512 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:37.589517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:37.589520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:37.589804 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:37.589932 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:37.590625 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 03:38:37.590855 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":198,"threads_started":1}
I20260926 03:38:37.592406 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:37.575325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:38043)
0926 03:38:37.576013 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:37.576050 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:37.577627 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:37.578190 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:37.578197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:37.578701 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:37.579268 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:37.579285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:37.580257 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:37.580261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:37.581483 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:37.581495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:37.582362 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:37.582369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:37.582603 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:37.583241 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:37.583273 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:37.585524 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:37.588448 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:37.588457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:37.588475 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:37.588872 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:37.589206 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:37.589210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:37.589214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:37.589410 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:37.589926 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:37.589934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:37.590268 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 03:38:37.590527 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 03:38:37.591075 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":233,"threads_started":1}
I20260926 03:38:37.593940 10421 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36665
I20260926 03:38:37.594229 10421 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:37.594944 10421 heartbeater.cc:511] Master 127.1.221.190:36665 requested a full tablet report, sending...
I20260926 03:38:37.596982 10232 ts_manager.cc:194] Registered new tserver with Master: 3f24f75635644a87904f374b15ed6742 (127.1.221.129:44017)
I20260926 03:38:37.598389 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:38043
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:38:37.628193 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:37.937166 10427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:37.937434 10427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:37.937528 10427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:37.944540 10427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:37.944651 10427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:37.944761 10427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:37.954336 10427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:37.955641 10427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:37.956869 10427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:37.969094 10435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:37.969188 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:37.969123 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:37.969985 10427 server_base.cc:1034] running on GCE node
I20260926 03:38:37.970542 10427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:37.971460 10427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:37.972674 10427 hybrid_clock.cc:648] HybridClock initialized: now 1790393917972615 us; error 51 us; skew 500 ppm
I20260926 03:38:37.972999 10427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:37 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:37.975947 10427 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:38:37.977300 10427 webserver.cc:460] Webserver started at http://127.1.221.130:34313/ using document root <none> and password file <none>
I20260926 03:38:37.977936 10427 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:37.978035 10427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:37.978331 10427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:37.981194 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "f75dbcc7753148d9acf01daef60b51fc"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "2f5bb03530451e3563a8c8ef0cbcceb6"
server_key_iv: "6ebde0b3883ceae77da2efe22c6f116d"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.981914 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "f75dbcc7753148d9acf01daef60b51fc"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "2f5bb03530451e3563a8c8ef0cbcceb6"
server_key_iv: "6ebde0b3883ceae77da2efe22c6f116d"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.986917 10427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:37.990345 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:37.991806 10427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:37.991947 10427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "f75dbcc7753148d9acf01daef60b51fc"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "2f5bb03530451e3563a8c8ef0cbcceb6"
server_key_iv: "6ebde0b3883ceae77da2efe22c6f116d"
server_key_version: "encryptionkey@0"
I20260926 03:38:37.992128 10427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:38.009516 10427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:38.013473 10427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:38.013703 10427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:38.014562 10427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:38.015911 10427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:38.015983 10427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:38.016072 10427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:38.016113 10427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:38.045392 10427 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38637
I20260926 03:38:38.045428 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38637 every 8 connection(s)
I20260926 03:38:38.046659 10427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:38.056556 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10427
I20260926 03:38:38.056718 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:38.057201 1910 external_mini_cluster.cc:1428] Setting key 05719a1f1a6f341f4982e2c52696e49c
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:38.066486 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.049128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49563 (local address 127.1.221.190:36665)
0926 03:38:38.049299 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:38.049304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:38.050450 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 03:38:38.052286 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:38.052291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:38.052380 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:38.052469 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:38.054117 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.054905 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.056524 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.057014 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.057129 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.060739 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:38.060745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:38.060748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:38.060790 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:38.063317 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.064142 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.064146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.064149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.064287 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.064734 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.064737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.064739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.065017 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:38.065167 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:38.065918 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 03:38:38.066154 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:38:38.067579 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.049574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:49563)
0926 03:38:38.050280 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:38.050319 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:38.052000 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:38.052740 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:38.052753 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:38.053333 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:38.053943 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:38.053963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.055064 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.055067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:38.056373 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:38.056386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.057253 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.057272 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:38.057528 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.058176 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:38.058207 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:38.060499 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:38.063499 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:38.063510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:38.063531 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:38.064001 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:38.064399 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:38.064404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:38.064408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:38.064615 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:38.065146 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:38.065154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:38.065544 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:38:38.065800 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 03:38:38.066382 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":254,"threads_started":1}
I20260926 03:38:38.069092 10555 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36665
I20260926 03:38:38.069450 10555 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:38.070308 10555 heartbeater.cc:511] Master 127.1.221.190:36665 requested a full tablet report, sending...
I20260926 03:38:38.072117 10232 ts_manager.cc:194] Registered new tserver with Master: f75dbcc7753148d9acf01daef60b51fc (127.1.221.130:38637)
I20260926 03:38:38.073172 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:49563
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:38:38.102121 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:38.415962 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:38.416240 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:38.416330 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:38.423506 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:38.423619 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:38.423774 10560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:38.435649 10560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:38.436877 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:38.438100 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:38.452373 10565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:38.452407 10566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:38.452664 10568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:38.453161 10560 server_base.cc:1034] running on GCE node
I20260926 03:38:38.453821 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:38.454847 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:38.456039 10560 hybrid_clock.cc:648] HybridClock initialized: now 1790393918455987 us; error 49 us; skew 500 ppm
I20260926 03:38:38.456339 10560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:38.459399 10560 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:38:38.460808 10560 webserver.cc:460] Webserver started at http://127.1.221.131:39041/ using document root <none> and password file <none>
I20260926 03:38:38.461478 10560 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:38.461588 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:38.461886 10560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:38.464891 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "322d92766d3c4fb9a7df97618f2df73e"
format_stamp: "Formatted at 2026-09-26 03:38:38 on dist-test-slave-lqrk"
server_key: "fa09e4bd3fd547730bb0a6480fd8b106"
server_key_iv: "b4e45b4d3949d23c4af1dd56938f12e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:38.465634 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "322d92766d3c4fb9a7df97618f2df73e"
format_stamp: "Formatted at 2026-09-26 03:38:38 on dist-test-slave-lqrk"
server_key: "fa09e4bd3fd547730bb0a6480fd8b106"
server_key_iv: "b4e45b4d3949d23c4af1dd56938f12e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:38.470852 10560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:38.474182 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:38.475521 10560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:38.475667 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "322d92766d3c4fb9a7df97618f2df73e"
format_stamp: "Formatted at 2026-09-26 03:38:38 on dist-test-slave-lqrk"
server_key: "fa09e4bd3fd547730bb0a6480fd8b106"
server_key_iv: "b4e45b4d3949d23c4af1dd56938f12e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:38.475852 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:38.510073 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:38.514417 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:38.514691 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:38.515668 10560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:38.517005 10560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:38.517081 10560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:38.517210 10560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:38.517262 10560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:38.549175 10560 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32775
I20260926 03:38:38.549274 10687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32775 every 8 connection(s)
I20260926 03:38:38.550558 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:38.551961 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10560
I20260926 03:38:38.552109 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:38.552517 1910 external_mini_cluster.cc:1428] Setting key d023ce9715ff6d59219a8c6225f29b2c
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:38:38.572232 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.554703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46093 (local address 127.1.221.190:36665)
0926 03:38:38.554863 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:38.554868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:38.556265 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 03:38:38.558198 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:38.558203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:38.558289 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:38.558377 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:38.560000 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.560733 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.562233 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.562626 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.562749 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.566704 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:38.566709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:38.566713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:38.566745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:38.569334 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.569987 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.569991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.569993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.570101 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.570461 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.570465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.570468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.570846 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:38.570987 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:38.571662 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 03:38:38.571903 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:38:38.572968 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.555300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:46093)
0926 03:38:38.556115 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:38.556161 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:38.557933 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:38.558634 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:38.558643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:38.559175 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:38.559842 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:38.559860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.560886 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.560889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:38.562098 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:38.562110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.562861 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.562868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:38.563090 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.563817 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:38.563844 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:38.566480 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:38.569466 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:38.569473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:38.569486 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:38.569891 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:38.570173 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:38:38.570176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:38:38.570178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:38:38.570351 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:38:38.570952 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:38.570957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:38.571272 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 03:38:38.571529 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 03:38:38.571904 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260926 03:38:38.574231 10688 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36665
I20260926 03:38:38.574532 10688 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:38.575335 10688 heartbeater.cc:511] Master 127.1.221.190:36665 requested a full tablet report, sending...
I20260926 03:38:38.576843 10232 ts_manager.cc:194] Registered new tserver with Master: 322d92766d3c4fb9a7df97618f2df73e (127.1.221.131:32775)
I20260926 03:38:38.577600 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:46093
I20260926 03:38:38.580960 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
I20260926 03:38:38.601408 10421 heartbeater.cc:503] Master 127.1.221.190:36665 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260926 03:38:38.710248 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10293
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:38:38.725952 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.711429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.1.221.129:44017)
0926 03:38:38.712125 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:38.712133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:38.712160 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:38.712221 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:38.712227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:38.712449 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:38.712583 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:38.713808 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.714875 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.716098 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.716665 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.716759 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.720512 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:38.720525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:38.720541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:38.720588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:38.723495 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.724116 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.724123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.724125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.724248 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.724523 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.724526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.724528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.724903 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:38.725123 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:38.725331 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 03:38:38.725580 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":384,"threads_started":1}
I20260926 03:38:38.727372 10416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:38.807423 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10293
I20260926 03:38:38.820392 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10427
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:38:38.835145 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.821212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.1.221.130:38637)
0926 03:38:38.821689 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:38.821696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:38.821723 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:38.821812 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:38.821821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:38.822036 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:38.822170 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:38.823399 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.824385 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.825562 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.826116 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.826206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.830143 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:38.830157 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:38.830172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:38.830207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:38.832752 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.833376 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.833383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.833385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.833502 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.833752 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.833755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.833757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.834079 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:38.834330 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:38.834593 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:38:38.834786 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
I20260926 03:38:38.836452 10550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:38.916522 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10427
I20260926 03:38:38.929208 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10560
Sep 26 03:38:38 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:38:38.944437 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:38.930075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.1.221.131:32775)
0926 03:38:38.930670 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:38.930677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:38.930704 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:38.930763 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:38.930770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:38.930977 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:38.931119 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:38.932438 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.933450 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.934674 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:38.935212 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:38.935332 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:38.939367 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:38.939401 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:38.939422 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:38.939463 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:38.941855 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.942457 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.942464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.942466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.942595 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:38.942935 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:38.942940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:38.942942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:38.943270 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:38.943586 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:38.943864 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:38:38.944114 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":295,"threads_started":1}
I20260926 03:38:38.945616 10683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:39.026908 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10560
I20260926 03:38:39.039175 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10200
I20260926 03:38:39.040354 10262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:39.142465 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10200
I20260926 03:38:39.158710 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36665
--webserver_interface=127.1.221.190
--webserver_port=43947
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:38:39.472075 10705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:39.472374 10705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:39.472505 10705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:39.479818 10705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:39.479938 10705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:39.480011 10705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:39.480085 10705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:39.480141 10705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:39.490123 10705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=43947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:39.491488 10705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:39.492771 10705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:39.504032 10710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:39.503986 10713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:39.504056 10711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:39.505151 10705 server_base.cc:1034] running on GCE node
I20260926 03:38:39.505951 10705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:39.507368 10705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:39.508602 10705 hybrid_clock.cc:648] HybridClock initialized: now 1790393919508549 us; error 44 us; skew 500 ppm
I20260926 03:38:39.508929 10705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:39 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393919, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:39.512019 10705 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:38:39.513497 10705 webserver.cc:460] Webserver started at http://127.1.221.190:43947/ using document root <none> and password file <none>
I20260926 03:38:39.514176 10705 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:39.514297 10705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:39.520062 10705 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:38:39.522866 10720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:39.524243 10705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:38:39.524417 10705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "911c2733835640c0b0e628b1fbc71ff2"
format_stamp: "Formatted at 2026-09-26 03:38:36 on dist-test-slave-lqrk"
server_key: "f16ea951bf1a8de3cf6265c860a1f6d4"
server_key_iv: "94e6d373e4a89f875d91ec19522bbe7d"
server_key_version: "encryptionkey@0"
I20260926 03:38:39.525388 10705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:39.548717 10705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:39.552776 10705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:39.553049 10705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:39.570348 10705 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36665
I20260926 03:38:39.570369 10771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36665 every 8 connection(s)
I20260926 03:38:39.571930 10705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:39.577953 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10705
I20260926 03:38:39.578950 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:44017
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:36665
--webserver_port=34041
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:38:39.579557 10772 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:38:39.584118 10772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Bootstrap starting.
I20260926 03:38:39.603740 10772 log.cc:824] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:39.613142 10772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:38:39.613629 10772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Bootstrap complete.
I20260926 03:38:39.622444 10772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:39.623294 10772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911c2733835640c0b0e628b1fbc71ff2, State: Initialized, Role: FOLLOWER
I20260926 03:38:39.623901 10772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:39.624125 10772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:39.624231 10772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:39.624385 10772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:38:39.628165 10772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:39.628634 10772 leader_election.cc:302] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911c2733835640c0b0e628b1fbc71ff2; no voters:
I20260926 03:38:39.629245 10772 leader_election.cc:288] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:38:39.629388 10776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:38:39.630486 10776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [term 2 LEADER]: Becoming Leader. State: Replica: 911c2733835640c0b0e628b1fbc71ff2, State: Running, Role: LEADER
I20260926 03:38:39.631090 10776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } }
I20260926 03:38:39.631398 10772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:39.633451 10778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911c2733835640c0b0e628b1fbc71ff2. Latest consensus state: current_term: 2 leader_uuid: "911c2733835640c0b0e628b1fbc71ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } } }
I20260926 03:38:39.633807 10778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:39.634310 10777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "911c2733835640c0b0e628b1fbc71ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c2733835640c0b0e628b1fbc71ff2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36665 } } }
I20260926 03:38:39.634601 10777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:39.637976 10783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:39.644081 10783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:39.645625 10783 catalog_manager.cc:1259] Loaded cluster ID: ef6713104c4b44339e4933f4fef4cd53
I20260926 03:38:39.645790 10783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:39.650115 10783 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:39.651721 10783 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 911c2733835640c0b0e628b1fbc71ff2: Loaded TSK: 0
I20260926 03:38:39.653398 10783 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:38:39.952942 10774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:39.953229 10774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:39.953361 10774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:39.960701 10774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:39.960824 10774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:39.960976 10774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:39.970750 10774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:44017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=34041
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:39.972107 10774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:39.973410 10774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:39.991952 10798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:39.991940 10799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:39.992997 10774 server_base.cc:1034] running on GCE node
W20260926 03:38:39.993157 10801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:39.993836 10774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:39.994831 10774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:39.996047 10774 hybrid_clock.cc:648] HybridClock initialized: now 1790393919995977 us; error 60 us; skew 500 ppm
I20260926 03:38:39.996372 10774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:39 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393919, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:39.999637 10774 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260926 03:38:40.001148 10774 webserver.cc:460] Webserver started at http://127.1.221.129:34041/ using document root <none> and password file <none>
I20260926 03:38:40.001849 10774 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:40.001966 10774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:40.007540 10774 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:38:40.010461 10808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.012091 10774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:40.012257 10774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "3f24f75635644a87904f374b15ed6742"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "7d11d339f2c32f43f1976fd6119a4589"
server_key_iv: "7dfa46d5757d4fdc599b587a529cf010"
server_key_version: "encryptionkey@0"
I20260926 03:38:40.013520 10774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:40.030037 10774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:40.034127 10774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:40.034382 10774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:40.035372 10774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:40.036767 10774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:40.036860 10774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.036963 10774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:40.037034 10774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.066728 10774 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:44017
I20260926 03:38:40.066787 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:44017 every 8 connection(s)
I20260926 03:38:40.068279 10774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:40.073339 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10774
I20260926 03:38:40.074065 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:38637
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:36665
--webserver_port=34313
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:38:40.395445 10925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:40.395758 10925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:40.395895 10925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:40.403318 10925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:40.403492 10925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:40.403654 10925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:40.413632 10925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:38637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=34313
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:40.415016 10925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:40.416395 10925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:40.434428 10932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:40.436049 10931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:40.437204 10934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:40.437466 10925 server_base.cc:1034] running on GCE node
I20260926 03:38:40.438517 10925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:40.439762 10925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:40.441051 10925 hybrid_clock.cc:648] HybridClock initialized: now 1790393920440976 us; error 82 us; skew 500 ppm
I20260926 03:38:40.441388 10925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393920, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:40.444957 10925 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260926 03:38:40.446602 10925 webserver.cc:460] Webserver started at http://127.1.221.130:34313/ using document root <none> and password file <none>
I20260926 03:38:40.447327 10925 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:40.447482 10925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:40.452996 10925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:38:40.455684 10941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.457060 10925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:38:40.457206 10925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "f75dbcc7753148d9acf01daef60b51fc"
format_stamp: "Formatted at 2026-09-26 03:38:37 on dist-test-slave-lqrk"
server_key: "2f5bb03530451e3563a8c8ef0cbcceb6"
server_key_iv: "6ebde0b3883ceae77da2efe22c6f116d"
server_key_version: "encryptionkey@0"
I20260926 03:38:40.458122 10925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:40.481048 10925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:40.485181 10925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:40.485431 10925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:40.486460 10925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:40.487886 10925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:40.487972 10925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.488078 10925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:40.488142 10925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.526966 10925 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38637
I20260926 03:38:40.527010 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38637 every 8 connection(s)
I20260926 03:38:40.528491 10925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:40.531488 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 10925
I20260926 03:38:40.532166 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:32775
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:36665
--webserver_port=39041
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:38:40.872414 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:40.872717 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:40.872857 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:40.880177 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:40.880307 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:40.880486 11058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:40.890509 11058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:32775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=39041
--tserver_master_addrs=127.1.221.190:36665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:40.891937 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:40.893213 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:40.909024 11065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:40.909399 11064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:40.921023 11067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:40.921717 11058 server_base.cc:1034] running on GCE node
I20260926 03:38:40.922712 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:40.924264 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:40.925683 11058 hybrid_clock.cc:648] HybridClock initialized: now 1790393920925596 us; error 77 us; skew 500 ppm
I20260926 03:38:40.926153 11058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:38:40 dist-test-slave-lqrk krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393920, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:38:40.931354 11058 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260926 03:38:40.933967 11058 webserver.cc:460] Webserver started at http://127.1.221.131:39041/ using document root <none> and password file <none>
I20260926 03:38:40.935114 11058 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:40.935308 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:40.944517 11058 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:38:40.948964 11074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:40.951131 11058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:38:40.951355 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "322d92766d3c4fb9a7df97618f2df73e"
format_stamp: "Formatted at 2026-09-26 03:38:38 on dist-test-slave-lqrk"
server_key: "fa09e4bd3fd547730bb0a6480fd8b106"
server_key_iv: "b4e45b4d3949d23c4af1dd56938f12e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:40.952987 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:40.990118 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:40.996387 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:40.996750 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:40.998172 11058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:41.000186 11058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:41.000314 11058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:41.000428 11058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:41.000506 11058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:41.044064 11058 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32775
I20260926 03:38:41.044247 11186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32775 every 8 connection(s)
I20260926 03:38:41.046177 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:41.048714 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11058
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:41.101270 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.049945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.1.221.190:36665)
0926 03:38:41.075773 (+ 25828us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.075779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.079189 (+ 3410us) server_negotiation.cc:407] Connection header received
0926 03:38:41.079403 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.079409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.079520 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.079657 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.082171 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.083270 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.085321 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.085997 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.086133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.090011 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:41.090024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:41.090052 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:41.090112 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:41.094437 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:41.095336 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:41.095346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:41.095359 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:41.095631 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:38:41.096347 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:38:41.096353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:38:41.096356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:38:41.097059 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:41.097370 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:41.098221 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 03:38:41.099473 (+ 1252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25708,"thread_start_us":25582,"threads_started":1}
W20260926 03:38:41.264638 10924 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.182s user 0.008s sys 0.008s
W20260926 03:38:41.266552 10923 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:40.071224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:46513 (local address 127.1.221.190:36665)
0926 03:38:40.072084 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:40.072101 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:40.072351 (+ 250us) server_negotiation.cc:407] Connection header received
0926 03:38:40.074990 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:40.075004 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:40.075362 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:40.075726 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:40.077502 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.078752 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.080870 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.081928 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.082095 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.266155 (+1184060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:46513: BlockingRecv error: recv got EOF from 127.1.221.129:46513 (error 108)
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":260,"threads_started":1}
I20260926 03:38:41.267550 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:40.071349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:46513)
0926 03:38:40.072178 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:40.072229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:40.074406 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:40.075917 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:40.075927 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:40.076607 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:40.077322 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:40.077345 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.078933 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.078937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:40.080728 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:40.080742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.082130 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.082134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:40.082342 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:40.082974 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:40.082997 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.265736 (+1182739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":316,"threads_started":1}
W20260926 03:38:41.268730 10921 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:36665 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:36665: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:38:41.282812 10923 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:41.270213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53653 (local address 127.1.221.190:36665)
0926 03:38:41.270416 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.270423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.270443 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:38:41.270506 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.270512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.270594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.270691 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.271892 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.273153 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.275094 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.275801 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.275926 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.282510 (+ 6584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:53653: BlockingRecv error: recv got EOF from 127.1.221.129:53653 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:38:41.283625 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.269559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:53653)
0926 03:38:41.269788 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:41.269811 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:41.269952 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:41.270908 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:41.270914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:41.271482 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:41.271716 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.271730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.273310 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.273316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.274949 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.274963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.275976 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.275982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.276191 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.276957 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:41.276981 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.283277 (+ 6296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:41.296797 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.284628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:49283)
0926 03:38:41.284887 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:41.284909 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:41.285044 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:41.285433 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:41.285439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:41.285919 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:41.286162 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.286175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.287854 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.287861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.289572 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.289591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.290677 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.290684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.290891 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.291712 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:41.291739 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.296460 (+ 4721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260926 03:38:41.297482 10921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:38:41.297756 10923 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:41.284694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49283 (local address 127.1.221.190:36665)
0926 03:38:41.284880 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.284887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.284985 (+ 98us) server_negotiation.cc:407] Connection header received
0926 03:38:41.285187 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.285193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.285282 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.285387 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.286327 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.287675 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.289738 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.290510 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.290647 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.297063 (+ 6416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:49283: BlockingRecv error: recv got EOF from 127.1.221.129:49283 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 03:38:41.694190 11057 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.133s user 0.006s sys 0.013s
W20260926 03:38:41.695641 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:40.531397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55579 (local address 127.1.221.190:36665)
0926 03:38:40.546700 (+ 15303us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:40.546708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:40.551207 (+ 4499us) server_negotiation.cc:407] Connection header received
0926 03:38:40.551303 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:40.551309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:40.551453 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:40.551571 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:40.554367 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.555566 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.558384 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.559071 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.559210 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.695363 (+1136153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:55579: BlockingRecv error: recv got EOF from 127.1.221.130:55579 (error 108)
Metrics: {"server-negotiator.queue_time_us":15123,"thread_start_us":15305,"threads_started":1}
I20260926 03:38:41.696589 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:40.532039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:55579)
0926 03:38:40.547652 (+ 15613us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:40.547740 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:40.550675 (+ 2935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:40.552167 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:40.552183 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:40.553128 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:40.554121 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:40.554150 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.555757 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.555762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:40.558168 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:40.558188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:40.559356 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:40.559363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:40.559778 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:40.560658 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:40.560709 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.695235 (+1134526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15125,"thread_start_us":14863,"threads_started":1}
W20260926 03:38:41.697592 11054 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:36665 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:36665: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:41.708130 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.698156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:60121)
0926 03:38:41.698379 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:41.698400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:41.698545 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:41.698871 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:41.698876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:41.699404 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:41.699643 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.699655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.700722 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.700728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.702359 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.702374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.703625 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.703631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.703779 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.704388 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:41.704406 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.707850 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:38:41.708267 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:41.698274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:60121 (local address 127.1.221.190:36665)
0926 03:38:41.698400 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.698405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.698460 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:38:41.698658 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.698673 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.698746 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.698824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.699770 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.700562 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.702505 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.703115 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.703226 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.708049 (+ 4823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:60121: BlockingRecv error: recv got EOF from 127.1.221.130:60121 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:41 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:41.717840 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.708868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:57801)
0926 03:38:41.709009 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:41.709037 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:41.709134 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:41.709615 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:41.709618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:41.709879 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:41.710024 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.710031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.711524 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.711528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.712459 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.712471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.713385 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.713389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.713510 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.714039 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:41.714056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:41.717591 (+ 3535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:38:41.718118 11054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:38:41.718139 11056 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:41.709131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57801 (local address 127.1.221.190:36665)
0926 03:38:41.709315 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.709322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.709341 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:38:41.709401 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.709406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.709485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.709596 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.710137 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.711363 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.712575 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.713250 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.713378 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.717850 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:57801: BlockingRecv error: recv got EOF from 127.1.221.130:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 03:38:42.111959 11191 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.015s user 0.008s sys 0.009s
W20260926 03:38:42.113224 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:41.075747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35557 (local address 127.1.221.190:36665)
0926 03:38:41.082711 (+ 6964us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:41.082719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:41.082741 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:38:41.082801 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:41.082807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:41.082892 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:41.083020 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:41.084818 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.087220 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.091237 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.095154 (+ 3917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.095750 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.112916 (+1017166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:35557: BlockingRecv error: recv got EOF from 127.1.221.131:35557 (error 108)
Metrics: {"server-negotiator.queue_time_us":6772,"thread_start_us":6605,"threads_started":1}
I20260926 03:38:42.113768 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:41.057933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:35557)
0926 03:38:41.074086 (+ 16153us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:41.074157 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:41.077007 (+ 2850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:41.083416 (+ 6409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:41.083431 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:41.084010 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:41.084618 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.084637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.087461 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.087468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.089774 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:41.091004 (+ 1230us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:41.095544 (+ 4540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:41.095552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:41.095938 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:41.096733 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:41.096767 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:42.112726 (+1015959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15703,"thread_start_us":19472,"threads_started":1}
W20260926 03:38:42.114579 11187 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:36665 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:36665: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:42.126487 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:42.115230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:44267)
0926 03:38:42.115439 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:42.115456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:42.115554 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:42.115899 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:42.115904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:42.116323 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:42.116526 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.116538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.117735 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.117739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.118676 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.118687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.119321 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.119325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.119483 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.122904 (+ 3421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:42.122923 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:42.126266 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:38:42.126679 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:42.115314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44267 (local address 127.1.221.190:36665)
0926 03:38:42.115549 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:42.115555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:42.115578 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:38:42.115665 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:42.115669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:42.115770 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:42.115856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:42.116660 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.117597 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.118799 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.119191 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.119280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.126475 (+ 7195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:44267: BlockingRecv error: recv got EOF from 127.1.221.131:44267 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:42.134934 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:42.127202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:54147)
0926 03:38:42.127349 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:42.127370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:42.127493 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:42.127822 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:42.127826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:42.128129 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:42.128302 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.128312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.129461 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.129466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.130446 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.130457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.131096 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.131101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.131288 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.131856 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:42.131872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:42.134732 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 03:38:42.135150 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:42.127281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54147 (local address 127.1.221.190:36665)
0926 03:38:42.127538 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:42.127542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:42.127558 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:38:42.127607 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:42.127610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:42.127688 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:42.127765 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:42.128426 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.129335 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.130567 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.130963 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.131065 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.134912 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:54147: BlockingRecv error: recv got EOF from 127.1.221.131:54147 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
W20260926 03:38:42.135203 11187 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:42.307600 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:42.298054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:58701)
0926 03:38:42.298594 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:42.298613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:42.298753 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:42.299086 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:42.299091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:42.299556 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:42.299704 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.299721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.300645 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.300649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.301626 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.301637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.302366 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.302372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.302524 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.303067 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:42.303087 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:42.307272 (+ 4185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":293,"threads_started":1}
W20260926 03:38:42.307715 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:42.298127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58701 (local address 127.1.221.190:36665)
0926 03:38:42.298319 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:42.298325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:42.298693 (+ 368us) server_negotiation.cc:407] Connection header received
0926 03:38:42.298855 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:42.298859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:42.298939 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:42.299037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:42.299830 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.300511 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.301795 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.302192 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.302307 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.307520 (+ 5213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:58701: BlockingRecv error: recv got EOF from 127.1.221.129:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:42 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:42.727613 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:42.718730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:47763)
0926 03:38:42.719288 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:42.719304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:42.719472 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:42.719803 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:42.719807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:42.720262 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:42.720408 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.720417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.721573 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.721577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.722619 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:42.722632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.723466 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.723474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:42.723624 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.724157 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:42.724175 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:42.727320 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":300,"threads_started":1}
W20260926 03:38:42.727787 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:42.718963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:47763 (local address 127.1.221.190:36665)
0926 03:38:42.719174 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:42.719180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:42.719407 (+ 227us) server_negotiation.cc:407] Connection header received
0926 03:38:42.719599 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:42.719603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:42.719665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:42.719756 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:42.720549 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.721433 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.722757 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:42.723262 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:42.723372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:42.727572 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:47763: BlockingRecv error: recv got EOF from 127.1.221.130:47763 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:43.144479 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:43.135849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:44981)
0926 03:38:43.136416 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:43.136435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:43.136577 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:43.136897 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:43.136902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:43.137413 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:43.137554 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.137563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.138585 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.138591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.139600 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.139611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.140359 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.140365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.140519 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.141070 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:43.141087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:43.144246 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
W20260926 03:38:43.144601 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:43.135995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44981 (local address 127.1.221.190:36665)
0926 03:38:43.136211 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:43.136218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:43.136519 (+ 301us) server_negotiation.cc:407] Connection header received
0926 03:38:43.136695 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:43.136700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:43.136774 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:43.136855 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:43.137685 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.138464 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.139731 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.140198 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.140303 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.144404 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:44981: BlockingRecv error: recv got EOF from 127.1.221.131:44981 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:43.319954 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:43.308552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:48933)
0926 03:38:43.311747 (+ 3195us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:43.311764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:43.311904 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:43.312223 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:43.312228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:43.312641 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:43.312819 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.312827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.313827 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.313832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.314750 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.314761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.315444 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.315450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.315614 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.316179 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:43.316215 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:43.319699 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3085,"thread_start_us":2972,"threads_started":1}
W20260926 03:38:43.320158 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:43.308641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:48933 (local address 127.1.221.190:36665)
0926 03:38:43.308897 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:43.308903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:43.311853 (+ 2950us) server_negotiation.cc:407] Connection header received
0926 03:38:43.312020 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:43.312026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:43.312093 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:43.312181 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:43.312957 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.313695 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.314882 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.315282 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.315406 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.319936 (+ 4530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:48933: BlockingRecv error: recv got EOF from 127.1.221.129:48933 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:43 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:43.737088 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:43.728552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:51913)
0926 03:38:43.729043 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:43.729058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:43.729196 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:43.729504 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:43.729506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:43.729891 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:43.730071 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.730079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.731118 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.731122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.732088 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:43.732100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.732796 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.732801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:43.732922 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.733458 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:43.733475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:43.736854 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":289,"threads_started":1}
W20260926 03:38:43.737202 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:43.728653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51913 (local address 127.1.221.190:36665)
0926 03:38:43.728811 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:43.728816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:43.729152 (+ 336us) server_negotiation.cc:407] Connection header received
0926 03:38:43.729330 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:43.729334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:43.729396 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:43.729481 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:43.730236 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.730970 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.732212 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:43.732648 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:43.732746 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:43.737017 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:51913: BlockingRecv error: recv got EOF from 127.1.221.130:51913 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:44.153640 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:44.145369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:47753)
0926 03:38:44.145890 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:44.145909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:44.146064 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:44.146436 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:44.146441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:44.146852 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:44.147033 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.147042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.148095 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.148100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.149000 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.149011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.149673 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.149677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.149789 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.150285 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:44.150302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:44.153392 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":286,"threads_started":1}
W20260926 03:38:44.153795 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:44.145533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:47753 (local address 127.1.221.190:36665)
0926 03:38:44.145716 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:44.145723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:44.145996 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:38:44.146191 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:44.146196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:44.146282 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:44.146380 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:44.147170 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.147975 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.149142 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.149557 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.149673 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.153581 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:47753: BlockingRecv error: recv got EOF from 127.1.221.131:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:44.329421 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:44.320832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:36151)
0926 03:38:44.321331 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:44.321346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:44.321464 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:44.321746 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:44.321749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:44.322089 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:44.322264 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.322273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.323267 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.323271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.324208 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.324221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.324897 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.324903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.325102 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.325690 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:44.325710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:44.329091 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":244,"threads_started":1}
W20260926 03:38:44.329550 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:44.320910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36151 (local address 127.1.221.190:36665)
0926 03:38:44.321070 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:44.321075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:44.321419 (+ 344us) server_negotiation.cc:407] Connection header received
0926 03:38:44.321556 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:44.321560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:44.321623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:44.321718 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:44.322382 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.323134 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.324317 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.324751 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.324845 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.329316 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:36151: BlockingRecv error: recv got EOF from 127.1.221.129:36151 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:44 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:44.746627 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:44.737996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:48235)
0926 03:38:44.738592 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:44.738613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:44.738758 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:44.739061 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:44.739065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:44.739458 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:44.739641 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.739651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.740637 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.740641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.741581 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:44.741594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.742426 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.742432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:44.742607 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.743205 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:44.743223 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:44.746396 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":257,"threads_started":1}
W20260926 03:38:44.746815 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:44.738021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:48235 (local address 127.1.221.190:36665)
0926 03:38:44.738171 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:44.738185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:44.738699 (+ 514us) server_negotiation.cc:407] Connection header received
0926 03:38:44.738862 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:44.738866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:44.738926 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:44.739005 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:44.739769 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.740481 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.741727 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:44.742246 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:44.742363 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:44.746585 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:48235: BlockingRecv error: recv got EOF from 127.1.221.130:48235 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:45.163355 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:45.154520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:51223)
0926 03:38:45.155027 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:45.155043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:45.155154 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:45.155443 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:45.155446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:45.155819 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:45.155987 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.155995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.156971 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.156976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.157882 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.157894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.158503 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.158507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.158630 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.159312 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:45.159329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:45.163046 (+ 3717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":246,"threads_started":1}
W20260926 03:38:45.163481 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:45.154624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51223 (local address 127.1.221.190:36665)
0926 03:38:45.154785 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:45.154789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:45.155108 (+ 319us) server_negotiation.cc:407] Connection header received
0926 03:38:45.155242 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:45.155245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:45.155319 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:45.155428 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:45.156099 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.156797 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.157995 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.158385 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.158468 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.163254 (+ 4786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:51223: BlockingRecv error: recv got EOF from 127.1.221.131:51223 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:45.338584 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:45.330539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:60673)
0926 03:38:45.330970 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:45.330986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:45.331094 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:45.331403 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:45.331407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:45.331771 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:45.331932 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.331941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.332957 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.332961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.333943 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.333954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.334576 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.334580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.334698 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.335259 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:45.335281 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:45.338320 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
W20260926 03:38:45.338747 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:45.330589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60673 (local address 127.1.221.190:36665)
0926 03:38:45.330799 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:45.330804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:45.331054 (+ 250us) server_negotiation.cc:407] Connection header received
0926 03:38:45.331178 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:45.331182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:45.331263 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:45.331339 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:45.332045 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.332859 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.334066 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.334463 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.334553 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.338521 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:60673: BlockingRecv error: recv got EOF from 127.1.221.129:60673 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:45 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:45.756155 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:45.747691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:55769)
0926 03:38:45.748257 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:45.748276 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:45.748400 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:45.748787 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:45.748792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:45.749179 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:45.749350 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.749359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.750269 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.750273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.751246 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:45.751257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.751920 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.751924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:45.752054 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.752583 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:45.752599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:45.755804 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":295,"threads_started":1}
W20260926 03:38:45.756242 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:45.747787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55769 (local address 127.1.221.190:36665)
0926 03:38:45.747962 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:45.747968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:45.748353 (+ 385us) server_negotiation.cc:407] Connection header received
0926 03:38:45.748524 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:45.748529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:45.748630 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:45.748731 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:45.749469 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.750173 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.751361 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:45.751797 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:45.751887 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:45.756025 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:55769: BlockingRecv error: recv got EOF from 127.1.221.130:55769 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:46.172926 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:46.164482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:44631)
0926 03:38:46.164989 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:46.165006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:46.165130 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:46.165426 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:46.165429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:46.165790 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:46.165943 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.165950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.166912 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.166915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.167967 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.167984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.168774 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.168780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.168945 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.169533 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:46.169560 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:46.172682 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":265,"threads_started":1}
W20260926 03:38:46.173115 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:46.164560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44631 (local address 127.1.221.190:36665)
0926 03:38:46.164724 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:46.164729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:46.165077 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:38:46.165237 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:46.165241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:46.165303 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:46.165389 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:46.166056 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.166799 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.168099 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.168621 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.168734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.172886 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:44631: BlockingRecv error: recv got EOF from 127.1.221.131:44631 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:46.347316 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:46.339471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:44371)
0926 03:38:46.339914 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:46.339929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:46.340057 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:46.340316 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:46.340319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:46.340728 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:46.340870 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.340879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.341809 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.341812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.342718 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.342728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.343280 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.343284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.343420 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.344131 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:46.344148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:46.347094 (+ 2946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
W20260926 03:38:46.347548 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:46.339532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44371 (local address 127.1.221.190:36665)
0926 03:38:46.339748 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:46.339754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:46.340003 (+ 249us) server_negotiation.cc:407] Connection header received
0926 03:38:46.340147 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:46.340150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:46.340212 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:46.340289 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:46.340998 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.341714 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.342819 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.343179 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.343261 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.347285 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:44371: BlockingRecv error: recv got EOF from 127.1.221.129:44371 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:46 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:46.765692 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:46.757147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:53245)
0926 03:38:46.757689 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:46.757705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:46.757816 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:46.758134 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:46.758139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:46.758548 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:46.758715 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.758724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.759787 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.759791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.760723 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:46.760735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.761432 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.761438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:46.761606 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.762142 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:46.762159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:46.765379 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":294,"threads_started":1}
W20260926 03:38:46.765829 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:46.757292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:53245 (local address 127.1.221.190:36665)
0926 03:38:46.757495 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:46.757501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:46.757776 (+ 275us) server_negotiation.cc:407] Connection header received
0926 03:38:46.757911 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:46.757914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:46.757976 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:46.758065 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:46.758854 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.759637 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.760856 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:46.761305 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:46.761415 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:46.765585 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:53245: BlockingRecv error: recv got EOF from 127.1.221.130:53245 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:47.186098 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:47.173784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:37617)
0926 03:38:47.174304 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:47.174326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:47.174460 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:47.174798 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:47.174802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:47.175248 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:47.175493 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.175508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.176695 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.176700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.177623 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.177635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.178273 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.178278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.178413 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.182658 (+ 4245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:47.182680 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:47.185848 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
W20260926 03:38:47.186398 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:47.173916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37617 (local address 127.1.221.190:36665)
0926 03:38:47.174134 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:47.174140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:47.174488 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:38:47.174589 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:47.174593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:47.174655 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:47.174742 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:47.175668 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.176562 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.177732 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.178154 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.178249 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.186148 (+ 7899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:37617: BlockingRecv error: recv got EOF from 127.1.221.131:37617 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:47.356426 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:47.348274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:58025)
0926 03:38:47.348702 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:47.348717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:47.348828 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:47.349085 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:47.349088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:47.349467 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:47.349627 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.349635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.350527 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.350530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.351484 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.351497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.352178 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.352182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.352304 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.352826 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:47.352842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:47.356112 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
W20260926 03:38:47.356590 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:47.348354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58025 (local address 127.1.221.190:36665)
0926 03:38:47.348568 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:47.348574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:47.348791 (+ 217us) server_negotiation.cc:407] Connection header received
0926 03:38:47.348916 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:47.348919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:47.348981 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:47.349056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:47.349729 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.350430 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.351611 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.352040 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.352147 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.356351 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:58025: BlockingRecv error: recv got EOF from 127.1.221.129:58025 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:47 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:47.775094 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:47.766760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:38237)
0926 03:38:47.767234 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:47.767250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:47.767401 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:47.767735 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:47.767740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:47.768136 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:47.768295 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.768304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.769264 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.769268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.770176 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:47.770187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.770873 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.770879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:47.771045 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.771666 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:47.771684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:47.774794 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":270,"threads_started":1}
W20260926 03:38:47.775211 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:47.766846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:38237 (local address 127.1.221.190:36665)
0926 03:38:47.767021 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:47.767027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:47.767332 (+ 305us) server_negotiation.cc:407] Connection header received
0926 03:38:47.767523 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:47.767527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:47.767590 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:47.767675 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:47.768436 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.769131 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.770312 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:47.770726 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:47.770833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:47.774996 (+ 4163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:38237: BlockingRecv error: recv got EOF from 127.1.221.130:38237 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:48.195266 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:48.186987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:57651)
0926 03:38:48.187502 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:48.187521 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:48.187646 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:48.187945 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:48.187950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:48.188451 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:48.188615 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.188624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.189631 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.189637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.190533 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.190543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.191202 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.191208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.191403 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.191927 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:48.191944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:48.194946 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":254,"threads_started":1}
W20260926 03:38:48.195410 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:48.187069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57651 (local address 127.1.221.190:36665)
0926 03:38:48.187256 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:48.187261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:48.187596 (+ 335us) server_negotiation.cc:407] Connection header received
0926 03:38:48.187755 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:48.187760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:48.187823 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:48.187899 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:48.188753 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.189498 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.190666 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.191061 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.191165 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.195161 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:57651: BlockingRecv error: recv got EOF from 127.1.221.131:57651 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:48.365825 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:48.357392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:52657)
0926 03:38:48.357945 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:48.357960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:48.358074 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:48.358331 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:48.358334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:48.358763 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:48.358935 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.358944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.359984 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.359988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.360995 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.361009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.361677 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.361681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.361791 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.362353 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:48.362369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:48.365582 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
W20260926 03:38:48.365907 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:48.357471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:52657 (local address 127.1.221.190:36665)
0926 03:38:48.357618 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:48.357622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:48.358025 (+ 403us) server_negotiation.cc:407] Connection header received
0926 03:38:48.358161 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:48.358164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:48.358224 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:48.358303 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:48.359064 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.359873 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.361129 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.361559 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.361648 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.365735 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:52657: BlockingRecv error: recv got EOF from 127.1.221.129:52657 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:48 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:48.784416 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:48.776042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:47651)
0926 03:38:48.776557 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:48.776572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:48.776687 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:48.776954 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:48.776957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:48.777333 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:48.777497 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.777504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.778418 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.778421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.779355 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:48.779365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.780051 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.780055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:48.780166 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.780695 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:48.780710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:48.784088 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":226,"threads_started":1}
W20260926 03:38:48.784533 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:48.776102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:47651 (local address 127.1.221.190:36665)
0926 03:38:48.776296 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:48.776303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:48.776647 (+ 344us) server_negotiation.cc:407] Connection header received
0926 03:38:48.776782 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:48.776785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:48.776846 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:48.776925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:48.777613 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.778319 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.779508 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:48.779926 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:48.780015 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:48.784280 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:47651: BlockingRecv error: recv got EOF from 127.1.221.130:47651 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:49.204705 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:49.196185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:34603)
0926 03:38:49.196740 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:49.196756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:49.196872 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:49.197190 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:49.197193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:49.197576 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:49.197740 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.197748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.198868 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.198874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.200085 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.200098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.200792 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.200798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.200953 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.201452 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:49.201481 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:49.204456 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":246,"threads_started":1}
W20260926 03:38:49.204833 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:49.196281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:34603 (local address 127.1.221.190:36665)
0926 03:38:49.196471 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:49.196477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:49.196820 (+ 343us) server_negotiation.cc:407] Connection header received
0926 03:38:49.196981 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:49.196987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:49.197070 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:49.197178 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:49.197890 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.198716 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.200213 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.200653 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.200758 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.204652 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:34603: BlockingRecv error: recv got EOF from 127.1.221.131:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:49.375051 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:49.366722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:50555)
0926 03:38:49.367445 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:49.367464 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:49.367614 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:49.367959 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:49.367965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:49.368362 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:49.368550 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.368564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.369595 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.369600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.370516 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.370537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.371214 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.371218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.371336 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.371886 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:49.371903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:49.374801 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":391,"threads_started":1}
W20260926 03:38:49.375306 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:49.366958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:50555 (local address 127.1.221.190:36665)
0926 03:38:49.367199 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:49.367205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:49.367576 (+ 371us) server_negotiation.cc:407] Connection header received
0926 03:38:49.367729 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:49.367734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:49.367813 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:49.367904 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:49.368665 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.369458 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.370647 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.371075 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.371173 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.375049 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:50555: BlockingRecv error: recv got EOF from 127.1.221.129:50555 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:49 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:49.797852 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:49.785559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:51731)
0926 03:38:49.786097 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:49.786113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:49.786273 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:49.786637 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:49.786642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:49.787062 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:49.787226 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.787235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.788484 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.788490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.789821 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:49.789836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.790629 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.790634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:49.790748 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.794127 (+ 3379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:49.794146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:49.797548 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
W20260926 03:38:49.798107 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:49.785736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51731 (local address 127.1.221.190:36665)
0926 03:38:49.785978 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:49.785985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:49.786202 (+ 217us) server_negotiation.cc:407] Connection header received
0926 03:38:49.786415 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:49.786419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:49.786482 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:49.786580 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:49.787401 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.788312 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.789964 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:49.790454 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:49.790557 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:49.797847 (+ 7290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:51731: BlockingRecv error: recv got EOF from 127.1.221.130:51731 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:50.214802 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:50.205697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:41227)
0926 03:38:50.206172 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:50.206188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:50.206319 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:50.206623 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:50.206628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:50.207021 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:50.207181 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.207190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.208534 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.208538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.209764 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.209777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.210477 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.210483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.210641 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.211192 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:50.211209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:50.214488 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":366,"threads_started":1}
W20260926 03:38:50.214942 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:50.205790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:41227 (local address 127.1.221.190:36665)
0926 03:38:50.206074 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:50.206080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:50.206271 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:38:50.206439 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:50.206443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:50.206503 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:50.206582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:50.207332 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.208303 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.209890 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.210307 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.210423 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.214699 (+ 4276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:41227: BlockingRecv error: recv got EOF from 127.1.221.131:41227 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:50.384570 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:50.376026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:39163)
0926 03:38:50.376585 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:50.376601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:50.376707 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:50.376986 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:50.376988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:50.377376 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:50.377531 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.377541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.378460 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.378463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.379445 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.379463 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.380123 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.380128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.380259 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.380774 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:50.380790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:50.384250 (+ 3460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
W20260926 03:38:50.384685 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:50.376082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39163 (local address 127.1.221.190:36665)
0926 03:38:50.376253 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:50.376258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:50.376675 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:38:50.376799 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:50.376803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:50.376881 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:50.376957 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:50.377641 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.378353 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.379574 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.379994 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.380085 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.384454 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:39163: BlockingRecv error: recv got EOF from 127.1.221.129:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:50 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:50.807919 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:50.798843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:52135)
0926 03:38:50.799416 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:50.799433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:50.799566 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:50.799886 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:50.799891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:50.800320 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:50.800462 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.800472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.801473 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.801478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.802546 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:50.802558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.803249 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.803255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:50.803461 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.804011 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:50.804029 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:50.807672 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
W20260926 03:38:50.808225 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:50.798915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52135 (local address 127.1.221.190:36665)
0926 03:38:50.799138 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:50.799144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:50.799514 (+ 370us) server_negotiation.cc:407] Connection header received
0926 03:38:50.799687 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:50.799692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:50.799755 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:50.799845 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:50.800599 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.801341 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.802676 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:50.803109 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:50.803214 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:50.807836 (+ 4622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:52135: BlockingRecv error: recv got EOF from 127.1.221.130:52135 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:51.224922 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:51.216098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:48821)
0926 03:38:51.216564 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:51.216579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:51.216707 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:51.217096 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:51.217101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:51.217497 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:51.217645 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.217655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.218794 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.218798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.219866 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.219880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.220585 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.220591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.220754 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.221294 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:51.221311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:51.224617 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
W20260926 03:38:51.225078 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:51.216189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:48821 (local address 127.1.221.190:36665)
0926 03:38:51.216387 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:51.216394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:51.216657 (+ 263us) server_negotiation.cc:407] Connection header received
0926 03:38:51.216897 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:51.216900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:51.216964 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:51.217048 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:51.217793 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.218665 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.219988 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.220444 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.220550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.224833 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:48821: BlockingRecv error: recv got EOF from 127.1.221.131:48821 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:51.393877 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:51.385503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:60475)
0926 03:38:51.386065 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:51.386086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:51.386237 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:51.386584 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:51.386590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:51.387051 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:51.387212 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.387221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.388233 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.388236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.389191 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.389203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.390043 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.390047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.390155 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.390679 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:51.390694 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:51.393623 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":303,"threads_started":1}
W20260926 03:38:51.394043 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:51.385593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60475 (local address 127.1.221.190:36665)
0926 03:38:51.385782 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:51.385788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:51.386185 (+ 397us) server_negotiation.cc:407] Connection header received
0926 03:38:51.386358 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:51.386364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:51.386436 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:51.386519 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:51.387345 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.388129 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.389320 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.389910 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.390023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.393805 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:60475: BlockingRecv error: recv got EOF from 127.1.221.129:60475 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:51 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:51.817287 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:51.808781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:35571)
0926 03:38:51.809244 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:51.809258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:51.809388 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:51.809721 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:51.809726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:51.810128 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:51.810283 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.810292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.811293 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.811298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.812273 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:51.812285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.813000 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.813007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:51.813158 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.813708 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:51.813726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:51.816892 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
W20260926 03:38:51.817390 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:51.808871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35571 (local address 127.1.221.190:36665)
0926 03:38:51.809080 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:51.809087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:51.809336 (+ 249us) server_negotiation.cc:407] Connection header received
0926 03:38:51.809512 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:51.809517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:51.809581 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:51.809660 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:51.810431 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.811155 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.812413 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:51.812833 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:51.812939 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:51.817134 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:35571: BlockingRecv error: recv got EOF from 127.1.221.130:35571 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:52.238178 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:52.226021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:50771)
0926 03:38:52.226490 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:52.226507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:52.226636 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:52.226948 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:52.226953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:52.227487 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:52.227697 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.227710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.228835 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.228841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.229884 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.229896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.230608 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.230614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.230773 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.234721 (+ 3948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:52.234739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:52.237925 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":216,"threads_started":1}
W20260926 03:38:52.238391 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:52.226201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50771 (local address 127.1.221.190:36665)
0926 03:38:52.226446 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:52.226452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:52.226575 (+ 123us) server_negotiation.cc:407] Connection header received
0926 03:38:52.226756 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:52.226762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:52.226823 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:52.226907 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:52.227822 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.228701 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.230013 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.230460 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.230568 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.238146 (+ 7578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:50771: BlockingRecv error: recv got EOF from 127.1.221.131:50771 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:52.403893 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:52.394810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:53791)
0926 03:38:52.395286 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:52.395302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:52.395446 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:52.395759 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:52.395762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:52.396171 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:52.396371 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.396380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.397584 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.397588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.398782 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.398801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.399602 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.399607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.399738 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.400278 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:52.400295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:52.403663 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
W20260926 03:38:52.404034 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:52.394913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53791 (local address 127.1.221.190:36665)
0926 03:38:52.395073 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:52.395078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:52.395371 (+ 293us) server_negotiation.cc:407] Connection header received
0926 03:38:52.395558 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:52.395561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:52.395622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:52.395706 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:52.396518 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.397424 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.398941 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.399461 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.399555 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.403841 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:53791: BlockingRecv error: recv got EOF from 127.1.221.129:53791 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:52 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:52.827206 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:52.818365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:40765)
0926 03:38:52.818875 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:52.818891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:52.819019 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:52.819333 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:52.819337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:52.819777 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:52.819990 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.820000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.821212 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.821216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.822396 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:52.822410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.823160 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.823164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:52.823293 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.823898 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:52.823916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:52.826985 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
W20260926 03:38:52.827502 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:52.818462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:40765 (local address 127.1.221.190:36665)
0926 03:38:52.818631 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:52.818636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:52.818964 (+ 328us) server_negotiation.cc:407] Connection header received
0926 03:38:52.819121 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:52.819124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:52.819193 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:52.819295 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:52.820135 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.821059 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.822535 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:52.822995 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:52.823120 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:52.827215 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:40765: BlockingRecv error: recv got EOF from 127.1.221.130:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:53.248103 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:53.239176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:46429)
0926 03:38:53.239691 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:53.239707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:53.239844 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:53.240158 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:53.240163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:53.240595 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:53.240838 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.240851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.241943 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.241947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.242848 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.242860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.243768 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.243772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.243902 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.244491 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:53.244510 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:53.247879 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
W20260926 03:38:53.248308 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:53.239292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46429 (local address 127.1.221.190:36665)
0926 03:38:53.239513 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:53.239519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:53.239792 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:38:53.239963 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:53.239968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:53.240030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:53.240116 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:53.240971 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.241815 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.242980 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.243481 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.243551 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.248084 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:46429: BlockingRecv error: recv got EOF from 127.1.221.131:46429 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:53.413733 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:53.404774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:60425)
0926 03:38:53.405324 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:53.405340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:53.405455 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:53.405737 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:53.405741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:53.406176 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:53.406366 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.406375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.407551 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.407555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.408489 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.408502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.409188 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.409193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.409323 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.409859 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:53.409876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:53.413401 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
W20260926 03:38:53.413859 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:53.404846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60425 (local address 127.1.221.190:36665)
0926 03:38:53.405034 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:53.405039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:53.405407 (+ 368us) server_negotiation.cc:407] Connection header received
0926 03:38:53.405558 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:53.405561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:53.405625 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:53.405710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:53.406580 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.407417 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.408612 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.409047 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.409138 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.413635 (+ 4497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:60425: BlockingRecv error: recv got EOF from 127.1.221.129:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:53 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:53.837117 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:53.828129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:59967)
0926 03:38:53.828709 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:53.828725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:53.828860 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:53.829191 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:53.829196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:53.829641 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:53.829813 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.829822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.830818 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.830823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.831855 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:53.831867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.832683 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.832689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:53.832855 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.833466 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:53.833485 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:53.836862 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
W20260926 03:38:53.837275 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:53.828210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59967 (local address 127.1.221.190:36665)
0926 03:38:53.828403 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:53.828408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:53.828809 (+ 401us) server_negotiation.cc:407] Connection header received
0926 03:38:53.828982 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:53.828986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:53.829056 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:53.829148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:53.829965 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.830681 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.831983 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:53.832543 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:53.832654 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:53.837057 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:59967: BlockingRecv error: recv got EOF from 127.1.221.130:59967 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:54.257894 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:54.249161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:43863)
0926 03:38:54.249702 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:54.249721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:54.249846 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:54.250173 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:54.250177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:54.250588 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:54.250776 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.250786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.251893 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.251897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.252826 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.252837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.253510 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.253514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.253640 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.254168 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:54.254184 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:54.257649 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
W20260926 03:38:54.258088 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:54.249241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:43863 (local address 127.1.221.190:36665)
0926 03:38:54.249443 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:54.249451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:54.249802 (+ 351us) server_negotiation.cc:407] Connection header received
0926 03:38:54.249956 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:54.249960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:54.250037 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:54.250112 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:54.250918 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.251732 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.252958 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.253367 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.253465 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.257848 (+ 4383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:43863: BlockingRecv error: recv got EOF from 127.1.221.131:43863 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:54.423585 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:54.414854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:39841)
0926 03:38:54.415368 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:54.415419 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:54.415553 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:54.415872 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:54.415877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:54.416293 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:54.416446 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.416455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.417473 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.417478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.418437 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.418453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.419152 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.419158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.419320 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.419948 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:54.419965 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:54.423224 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":286,"threads_started":1}
W20260926 03:38:54.423713 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:54.414953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39841 (local address 127.1.221.190:36665)
0926 03:38:54.415138 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:54.415145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:54.415503 (+ 358us) server_negotiation.cc:407] Connection header received
0926 03:38:54.415663 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:54.415667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:54.415747 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:54.415829 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:54.416591 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.417333 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.418581 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.419015 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.419122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.423482 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:39841: BlockingRecv error: recv got EOF from 127.1.221.129:39841 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:54 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:54.846758 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:54.837976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:54477)
0926 03:38:54.838477 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:54.838493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:54.838623 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:54.838941 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:54.838946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:54.839337 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:54.839539 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.839548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.840556 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.840561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.841471 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:54.841484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.842201 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.842207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:54.842369 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.842905 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:54.842922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:54.846426 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":292,"threads_started":1}
W20260926 03:38:54.846861 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:54.838132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:54477 (local address 127.1.221.190:36665)
0926 03:38:54.838351 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:54.838356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:54.838573 (+ 217us) server_negotiation.cc:407] Connection header received
0926 03:38:54.838744 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:54.838749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:54.838811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:54.838902 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:54.839684 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.840417 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.841620 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:54.842065 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:54.842174 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:54.846639 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:54477: BlockingRecv error: recv got EOF from 127.1.221.130:54477 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:55.267182 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:55.258805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.131:37511)
0926 03:38:55.259218 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:55.259234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:55.259346 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:55.259641 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:55.259643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:55.260032 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:55.260176 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.260184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.261210 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.261214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.262199 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.262211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.262897 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.262902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.263057 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.263687 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:55.263706 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:55.266959 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
W20260926 03:38:55.267347 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:55.258839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37511 (local address 127.1.221.190:36665)
0926 03:38:55.259025 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:55.259032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:55.259323 (+ 291us) server_negotiation.cc:407] Connection header received
0926 03:38:55.259478 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:55.259481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:55.259543 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:55.259619 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:55.260294 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.261081 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.262317 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.262711 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.262838 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.267138 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:37511: BlockingRecv error: recv got EOF from 127.1.221.131:37511 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:55.434264 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:55.425313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.129:34217)
0926 03:38:55.425803 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:55.425822 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:55.425938 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:55.426284 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:55.426287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:55.426648 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:55.426835 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.426844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.428134 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.428138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.429208 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.429222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.430027 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.430031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.430172 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.430771 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:55.430789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:55.434018 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
W20260926 03:38:55.434470 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:55.425445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34217 (local address 127.1.221.190:36665)
0926 03:38:55.425751 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:55.425755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:55.425894 (+ 139us) server_negotiation.cc:407] Connection header received
0926 03:38:55.426040 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:55.426044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:55.426153 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:55.426258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:55.426992 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.427981 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.429379 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.429901 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.430000 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.434203 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:34217: BlockingRecv error: recv got EOF from 127.1.221.129:34217 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:38:55 dist-test-slave-lqrk krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:38:55.856914 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:55.848144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36665 (local address 127.1.221.130:43539)
0926 03:38:55.848644 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:55.848660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:55.848781 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:55.849113 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:55.849116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:55.849458 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:55.849641 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.849650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.850755 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.850759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.851733 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:55.851746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.852482 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.852488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:55.852658 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.853245 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:38:55.853261 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:55.856675 (+ 3414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:36665: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
W20260926 03:38:55.857090 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:55.848261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43539 (local address 127.1.221.190:36665)
0926 03:38:55.848434 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:55.848438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:55.848734 (+ 296us) server_negotiation.cc:407] Connection header received
0926 03:38:55.848887 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:55.848891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:55.848989 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:55.849081 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:55.849804 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.850606 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.851872 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:55.852324 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:55.852444 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:55.856866 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:43539: BlockingRecv error: recv got EOF from 127.1.221.130:43539 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:38:56.058851 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10774
I20260926 03:38:56.071770 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:56.060114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.1.221.129:44017)
0926 03:38:56.060702 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:56.060709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:56.060737 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:38:56.060805 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:56.060812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:56.061061 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:56.061232 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:56.062865 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.063961 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.065214 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.065783 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.065915 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:56.068395 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:56.068405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:56.068420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:56.068458 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:56.071080 (+ 2622us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:38:56.071482 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39898: Request ticket server kudu/127.1.221.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":286,"threads_started":1}
W20260926 03:38:56.071887 11236 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39898: Request ticket server kudu/127.1.221.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:38:56.071884 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10774: Not authorized: Client connection negotiation failed: client connection to 127.1.221.129:44017: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:38:56.072104 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10774
I20260926 03:38:56.085640 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10925
I20260926 03:38:56.098562 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:56.086669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.1.221.130:38637)
0926 03:38:56.088007 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:56.088015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:56.088044 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:38:56.088112 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:56.088118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:56.088361 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:56.088560 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:56.089866 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.090953 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.092208 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.092784 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.092905 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:56.095289 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:56.095298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:56.095313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:56.095354 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:56.097876 (+ 2522us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:38:56.098213 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38782: Request ticket server kudu/127.1.221.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":1026,"threads_started":1}
W20260926 03:38:56.098680 11237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38782: Request ticket server kudu/127.1.221.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:38:56.098650 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10925: Not authorized: Client connection negotiation failed: client connection to 127.1.221.130:38637: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:38:56.098804 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10925
I20260926 03:38:56.112270 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11058
I20260926 03:38:56.124994 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:56.113489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.1.221.131:32775)
0926 03:38:56.114097 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:56.114104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:56.114132 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:38:56.114200 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:56.114206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:56.114454 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:56.114646 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:56.116152 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.117201 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.118511 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.119104 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.119233 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:56.121790 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:56.121800 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:56.121815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:38:56.121873 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:56.124302 (+ 2429us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:38:56.124658 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37514: Request ticket server kudu/127.1.221.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
W20260926 03:38:56.125101 11238 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37514: Request ticket server kudu/127.1.221.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:38:56.125061 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11058: Not authorized: Client connection negotiation failed: client connection to 127.1.221.131:32775: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.221.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:38:56.125255 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11058
I20260926 03:38:56.138996 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 10705
I20260926 03:38:56.140326 10767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:56.254000 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 10705
2026-09-26T03:38:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20906 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:56Z Disabled control of system clock
I20260926 03:38:56.322268 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37551
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:38:56.636090 11243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:56.636384 11243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:56.636464 11243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:56.643812 11243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:56.643918 11243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:56.643962 11243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:56.644016 11243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:56.644057 11243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:56.653293 11243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:37551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:56.654541 11243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:56.656348 11243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:56.667872 11251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:56.668025 11243 server_base.cc:1034] running on GCE node
W20260926 03:38:56.667876 11248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:56.667905 11249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:56.669055 11243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:56.670397 11243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:56.671592 11243 hybrid_clock.cc:648] HybridClock initialized: now 1790393936671529 us; error 51 us; skew 500 ppm
I20260926 03:38:56.671912 11243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:56.674010 11243 webserver.cc:460] Webserver started at http://127.1.221.190:44697/ using document root <none> and password file <none>
I20260926 03:38:56.674677 11243 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:56.674772 11243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:56.675057 11243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:56.677834 11243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "633904ff5bbe4436b3a8b18089b36357"
format_stamp: "Formatted at 2026-09-26 03:38:56 on dist-test-slave-lqrk"
server_key: "91165e6d9dbcb6531e88470c7cb43a7e"
server_key_iv: "f95249e14e6b3a31d3a46e3b494007e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:56.678613 11243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "633904ff5bbe4436b3a8b18089b36357"
format_stamp: "Formatted at 2026-09-26 03:38:56 on dist-test-slave-lqrk"
server_key: "91165e6d9dbcb6531e88470c7cb43a7e"
server_key_iv: "f95249e14e6b3a31d3a46e3b494007e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:56.683708 11243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:56.686991 11257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:56.688344 11243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:56.688494 11243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "633904ff5bbe4436b3a8b18089b36357"
format_stamp: "Formatted at 2026-09-26 03:38:56 on dist-test-slave-lqrk"
server_key: "91165e6d9dbcb6531e88470c7cb43a7e"
server_key_iv: "f95249e14e6b3a31d3a46e3b494007e4"
server_key_version: "encryptionkey@0"
I20260926 03:38:56.688688 11243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:56.704002 11243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:56.704806 11243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:56.705055 11243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:56.721544 11243 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:37551
I20260926 03:38:56.721568 11308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:37551 every 8 connection(s)
I20260926 03:38:56.722877 11243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:56.726471 11309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:56.730633 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11243
I20260926 03:38:56.730820 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:56.731660 1910 external_mini_cluster.cc:1428] Setting key bb3c7447b7969c7934a26d26569e1054
I20260926 03:38:56.737654 11309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357: Bootstrap starting.
I20260926 03:38:56.741108 11309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:56.742244 11309 log.cc:824] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:56.745361 11309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357: No bootstrap required, opened a new log
I20260926 03:38:56.745537 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:56.733658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.1.221.190:37551)
0926 03:38:56.734570 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:56.734583 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:56.734627 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:38:56.736368 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:56.736389 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:56.736850 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:56.737157 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:56.737833 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.739102 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.740879 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:56.741605 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:56.741766 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:56.743192 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:56.743206 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:56.743226 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:56.743886 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:56.743963 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:56.743983 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:56.744090 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:56.744300 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:38:56.744611 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":301,"threads_started":1}
I20260926 03:38:56.753214 11309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } }
I20260926 03:38:56.753613 11309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:56.753777 11309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633904ff5bbe4436b3a8b18089b36357, State: Initialized, Role: FOLLOWER
I20260926 03:38:56.754349 11309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } }
I20260926 03:38:56.754541 11309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:56.754632 11309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:56.754762 11309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:56.756840 11309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } }
I20260926 03:38:56.757296 11309 leader_election.cc:302] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 633904ff5bbe4436b3a8b18089b36357; no voters:
I20260926 03:38:56.757934 11309 leader_election.cc:288] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:56.758041 11314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:56.758724 11314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [term 1 LEADER]: Becoming Leader. State: Replica: 633904ff5bbe4436b3a8b18089b36357, State: Running, Role: LEADER
I20260926 03:38:56.759251 11314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } }
I20260926 03:38:56.759917 11309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:56.761711 11315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "633904ff5bbe4436b3a8b18089b36357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } } }
I20260926 03:38:56.762068 11315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:56.762499 11316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 633904ff5bbe4436b3a8b18089b36357. Latest consensus state: current_term: 1 leader_uuid: "633904ff5bbe4436b3a8b18089b36357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "633904ff5bbe4436b3a8b18089b36357" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37551 } } }
I20260926 03:38:56.762801 11316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357 [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:56.762900 11320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:56.768273 11320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:56.778100 11320 catalog_manager.cc:1347] Generated new cluster ID: 31e7d35ac3ff41a085bdd72e38dd8da2
I20260926 03:38:56.778209 11320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:56.791247 11320 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:56.792760 11320 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:56.807791 11320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 633904ff5bbe4436b3a8b18089b36357: Generated new TSK 0
I20260926 03:38:56.808668 11320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:56.819782 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--builtin_ntp_servers=127.1.221.148:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:38:57.135567 11333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:57.135841 11333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:57.135922 11333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:57.142987 11333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:57.143082 11333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:57.143193 11333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:57.151957 11333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:57.153178 11333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:57.154907 11333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:57.167598 11341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:57.167585 11339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:57.167678 11338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:57.168183 11333 server_base.cc:1034] running on GCE node
I20260926 03:38:57.168821 11333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:57.169729 11333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:57.170928 11333 hybrid_clock.cc:648] HybridClock initialized: now 1790393937170875 us; error 46 us; skew 500 ppm
I20260926 03:38:57.171243 11333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:57.173444 11333 webserver.cc:460] Webserver started at http://127.1.221.129:40487/ using document root <none> and password file <none>
I20260926 03:38:57.174115 11333 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:57.174211 11333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:57.174520 11333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:57.177299 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "0e062dda18bc4d3c9fc1667d0731a8bf"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "822d3ca9ae6c9596ae8b8ab67c365832"
server_key_iv: "e96d94f923e512b21aa4f4ebccf0fd93"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.178013 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "0e062dda18bc4d3c9fc1667d0731a8bf"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "822d3ca9ae6c9596ae8b8ab67c365832"
server_key_iv: "e96d94f923e512b21aa4f4ebccf0fd93"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.182934 11333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:57.186291 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.187638 11333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:57.187786 11333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "0e062dda18bc4d3c9fc1667d0731a8bf"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "822d3ca9ae6c9596ae8b8ab67c365832"
server_key_iv: "e96d94f923e512b21aa4f4ebccf0fd93"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.187970 11333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:57.215322 11333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:57.216240 11333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:57.216518 11333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:57.217460 11333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:57.218910 11333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:57.218995 11333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.219097 11333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:57.219173 11333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.248989 11333 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36541
I20260926 03:38:57.249058 11459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36541 every 8 connection(s)
I20260926 03:38:57.250228 11333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:57.260332 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11333
I20260926 03:38:57.260511 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:57.261147 1910 external_mini_cluster.cc:1428] Setting key a80716838446bfbc84a1a09c561c7218
I20260926 03:38:57.263980 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--builtin_ntp_servers=127.1.221.148:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:38:57.265574 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:57.252619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:40623 (local address 127.1.221.190:37551)
0926 03:38:57.253032 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:57.253037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:57.254148 (+ 1111us) server_negotiation.cc:407] Connection header received
0926 03:38:57.256175 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:57.256180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:57.256261 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:57.256358 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:57.257417 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.258137 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.259720 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.260147 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.260262 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:57.264776 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:57.264782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:57.264787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:57.264810 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:57.264865 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:57.264880 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:57.264977 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:57.265155 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:38:57.265255 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260926 03:38:57.266618 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:57.253084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37551 (local address 127.1.221.129:40623)
0926 03:38:57.253985 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:57.254032 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:57.255905 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:57.256597 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:57.256606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:57.256701 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:57.257245 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:57.257260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.258287 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.258291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:57.259590 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:57.259603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.260388 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.260395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:57.260655 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:57.264415 (+ 3760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:57.264418 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:57.264469 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:57.264556 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:57.264971 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:57.264977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:57.264983 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:38:57.265124 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:38:57.265460 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":244,"threads_started":1}
I20260926 03:38:57.268234 11460 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37551
I20260926 03:38:57.268514 11460 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:57.269297 11460 heartbeater.cc:511] Master 127.1.221.190:37551 requested a full tablet report, sending...
I20260926 03:38:57.271301 11274 ts_manager.cc:194] Registered new tserver with Master: 0e062dda18bc4d3c9fc1667d0731a8bf (127.1.221.129:36541)
I20260926 03:38:57.272935 11274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:40623
W20260926 03:38:57.578625 11464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:57.578908 11464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:57.579028 11464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:57.586344 11464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:57.586449 11464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:57.586594 11464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:57.595973 11464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:57.597265 11464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:57.599020 11464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:57.612058 11469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:57.612030 11470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:57.611908 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:57.612998 11464 server_base.cc:1034] running on GCE node
I20260926 03:38:57.613669 11464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:57.614660 11464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:57.615861 11464 hybrid_clock.cc:648] HybridClock initialized: now 1790393937615785 us; error 69 us; skew 500 ppm
I20260926 03:38:57.616185 11464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:57.618422 11464 webserver.cc:460] Webserver started at http://127.1.221.130:38589/ using document root <none> and password file <none>
I20260926 03:38:57.619118 11464 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:57.619218 11464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:57.619561 11464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:57.622391 11464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "0655bc8b271e471aa113a340c295c821"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "dfd2a4a6572c811abb762b78bdd78569"
server_key_iv: "980aefa715711e8d6b098d93b9e1a0ff"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.623154 11464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "0655bc8b271e471aa113a340c295c821"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "dfd2a4a6572c811abb762b78bdd78569"
server_key_iv: "980aefa715711e8d6b098d93b9e1a0ff"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.628258 11464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:38:57.631618 11478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.632941 11464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:38:57.633112 11464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "0655bc8b271e471aa113a340c295c821"
format_stamp: "Formatted at 2026-09-26 03:38:57 on dist-test-slave-lqrk"
server_key: "dfd2a4a6572c811abb762b78bdd78569"
server_key_iv: "980aefa715711e8d6b098d93b9e1a0ff"
server_key_version: "encryptionkey@0"
I20260926 03:38:57.633311 11464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:57.650884 11464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:57.651818 11464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:57.652132 11464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:57.653084 11464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:57.654496 11464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:57.654583 11464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.654692 11464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:57.654785 11464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:57.684917 11464 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36141
I20260926 03:38:57.684966 11590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36141 every 8 connection(s)
I20260926 03:38:57.686228 11464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:38:57.696046 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11464
I20260926 03:38:57.696218 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:38:57.696803 1910 external_mini_cluster.cc:1428] Setting key f5f88e8c7d06ab30915c015297fdaf43
I20260926 03:38:57.699149 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:57.688649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34051 (local address 127.1.221.190:37551)
0926 03:38:57.688866 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:57.688872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:57.690337 (+ 1465us) server_negotiation.cc:407] Connection header received
0926 03:38:57.692284 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:57.692289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:57.692360 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:57.692475 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:57.693686 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.694404 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.696019 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.696628 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.696766 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:57.698344 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:57.698349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:57.698352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:57.698370 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:57.698403 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:57.698415 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:57.698484 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:57.698738 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:38:57.698826 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:38:57.699877 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--builtin_ntp_servers=127.1.221.148:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:38:57.700342 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:57.689252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37551 (local address 127.1.221.130:34051)
0926 03:38:57.690171 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:57.690210 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:57.691995 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:57.692664 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:57.692673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:57.692771 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:57.693454 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:57.693472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.694531 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.694534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:57.695848 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:57.695865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:57.696866 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:57.696874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:57.697181 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:57.697974 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:57.697977 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:57.698034 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:57.698143 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:57.698508 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:57.698519 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:57.698527 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:38:57.698725 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 03:38:57.699126 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260926 03:38:57.701758 11591 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37551
I20260926 03:38:57.702081 11591 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:57.703034 11591 heartbeater.cc:511] Master 127.1.221.190:37551 requested a full tablet report, sending...
I20260926 03:38:57.704557 11274 ts_manager.cc:194] Registered new tserver with Master: 0655bc8b271e471aa113a340c295c821 (127.1.221.130:36141)
I20260926 03:38:57.705391 11274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:34051
W20260926 03:38:58.019701 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:58.019979 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:58.020061 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:58.027230 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:58.027335 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:58.027477 11594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:38:58.036695 11594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:58.037971 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:58.039796 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:58.052402 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:58.052362 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:58.052424 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:58.053040 11594 server_base.cc:1034] running on GCE node
I20260926 03:38:58.053697 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:58.054728 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:58.055924 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790393938055854 us; error 63 us; skew 500 ppm
I20260926 03:38:58.056242 11594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:58.058545 11594 webserver.cc:460] Webserver started at http://127.1.221.131:42679/ using document root <none> and password file <none>
I20260926 03:38:58.059262 11594 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:58.059356 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:58.059700 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:58.062705 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "c9ee5f2ecc0e4f04a242929cfcf6d2e3"
format_stamp: "Formatted at 2026-09-26 03:38:58 on dist-test-slave-lqrk"
server_key: "e8fac6d23612ca394c8a977f488edde8"
server_key_iv: "79ce55a25a0662bc68edcc58a48f9fd0"
server_key_version: "encryptionkey@0"
I20260926 03:38:58.063522 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "c9ee5f2ecc0e4f04a242929cfcf6d2e3"
format_stamp: "Formatted at 2026-09-26 03:38:58 on dist-test-slave-lqrk"
server_key: "e8fac6d23612ca394c8a977f488edde8"
server_key_iv: "79ce55a25a0662bc68edcc58a48f9fd0"
server_key_version: "encryptionkey@0"
I20260926 03:38:58.068730 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:38:58.072216 11608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:58.073670 11594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:38:58.073819 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "c9ee5f2ecc0e4f04a242929cfcf6d2e3"
format_stamp: "Formatted at 2026-09-26 03:38:58 on dist-test-slave-lqrk"
server_key: "e8fac6d23612ca394c8a977f488edde8"
server_key_iv: "79ce55a25a0662bc68edcc58a48f9fd0"
server_key_version: "encryptionkey@0"
I20260926 03:38:58.074007 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:58.122964 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:58.123878 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:58.124150 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:58.125146 11594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:58.126567 11594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:58.126654 11594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:58.126763 11594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:58.126829 11594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:58.156387 11594 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:33121
I20260926 03:38:58.156435 11720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:33121 every 8 connection(s)
I20260926 03:38:58.157648 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:38:58.163211 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11594
I20260926 03:38:58.163524 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:38:58.164059 1910 external_mini_cluster.cc:1428] Setting key c2d0ecf81c38e01366a0bd5562a4f7c2
I20260926 03:38:58.170814 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:58.160464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55783 (local address 127.1.221.190:37551)
0926 03:38:58.160662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:58.160666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:58.162221 (+ 1555us) server_negotiation.cc:407] Connection header received
0926 03:38:58.164143 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:58.164149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:58.164249 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:58.164380 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:58.165745 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.166665 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.168485 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.168905 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.169010 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:58.170190 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:58.170195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:58.170198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:58.170217 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:58.170248 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:58.170259 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:58.170335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:58.170495 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 03:38:58.170570 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:38:58.171945 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:58.161069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37551 (local address 127.1.221.131:55783)
0926 03:38:58.162051 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:58.162096 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:58.163828 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:58.164593 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:58.164604 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:58.164728 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:58.165512 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:58.165539 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.166844 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.166849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:58.168343 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:38:58.168355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.169083 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.169087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:38:58.169280 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:58.169883 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:58.169886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:58.169926 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:58.170011 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:58.170338 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:58.170344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:58.170349 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:38:58.170485 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:38:58.170814 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260926 03:38:58.173278 11721 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37551
I20260926 03:38:58.173599 11721 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:58.174322 11721 heartbeater.cc:511] Master 127.1.221.190:37551 requested a full tablet report, sending...
I20260926 03:38:58.175748 11274 ts_manager.cc:194] Registered new tserver with Master: c9ee5f2ecc0e4f04a242929cfcf6d2e3 (127.1.221.131:33121)
I20260926 03:38:58.176527 11274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:55783
I20260926 03:38:58.179677 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:38:58.185595 11462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:38:58.183980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.1.221.190:37551)
0926 03:38:58.184122 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:58.184126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:58.184615 (+ 489us) server_negotiation.cc:407] Connection header received
0926 03:38:58.184772 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:58.184778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:58.184863 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:58.184945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:58.185355 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60554: BlockingRecv error: recv got EOF from 127.0.0.1:60554 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:38:58.186494 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11333
I20260926 03:38:58.197822 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:58.187316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.1.221.129:36541)
0926 03:38:58.191456 (+ 4140us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:58.191465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:58.191497 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:38:58.191576 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:58.191583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:58.191828 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:58.192019 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:58.192470 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.193534 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.194841 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.195410 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.195515 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:58.196444 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:58.196451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:58.196465 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:58.197073 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:58.197121 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:58.197140 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:58.197222 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:58.197353 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:38:58.197462 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3942,"thread_start_us":3841,"threads_started":1}
I20260926 03:38:58.199081 11455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:58.278170 11460 heartbeater.cc:503] Master 127.1.221.190:37551 was elected leader, sending a full tablet report...
I20260926 03:38:58.279616 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11333
W20260926 03:38:58.292900 11259 connection.cc:535] server connection from 127.1.221.129:40623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:38:58.293216 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11464
I20260926 03:38:58.300524 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:58.294063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.1.221.130:36141)
0926 03:38:58.294597 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:58.294611 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:58.294638 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:38:58.294697 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:58.294704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:58.294934 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:58.295068 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:58.295437 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.296418 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.297577 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.298120 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.298213 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:58.299188 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:58.299197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:58.299212 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:58.299862 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:58.299910 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:58.299930 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:58.300012 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:58.300117 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:38:58.300242 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
I20260926 03:38:58.301731 11586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:58.382640 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11464
I20260926 03:38:58.395131 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11594
I20260926 03:38:58.402730 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:58.396045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.1.221.131:33121)
0926 03:38:58.396641 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:58.396648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:58.396678 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:38:58.396736 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:58.396742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:58.396977 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:58.397099 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:58.397494 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.398531 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.399725 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:38:58.400293 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:38:58.400388 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:38:58.401322 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:58.401330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:58.401344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:58.401941 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:58.401988 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:58.402007 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:58.402091 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:58.402258 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:38:58.402369 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":307,"threads_started":1}
I20260926 03:38:58.403973 11716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:58.486474 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11594
I20260926 03:38:58.498865 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11243
I20260926 03:38:58.500015 11304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:38:58.600745 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11243
2026-09-26T03:38:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2342 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:38:58Z Disabled control of system clock
I20260926 03:38:58.665709 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40825
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:38:58.979095 11740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:58.979372 11740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:58.979521 11740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:58.986713 11740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:38:58.986825 11740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:58.986864 11740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:58.986944 11740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:38:58.986999 11740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:38:58.996183 11740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:58.997488 11740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:58.999270 11740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:59.010468 11745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.010429 11746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.011148 11748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:59.011642 11740 server_base.cc:1034] running on GCE node
I20260926 03:38:59.012403 11740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:59.013835 11740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:59.015059 11740 hybrid_clock.cc:648] HybridClock initialized: now 1790393939015001 us; error 39 us; skew 500 ppm
I20260926 03:38:59.015447 11740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:59.017673 11740 webserver.cc:460] Webserver started at http://127.1.221.190:44637/ using document root <none> and password file <none>
I20260926 03:38:59.018424 11740 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:59.018512 11740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:59.018780 11740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:59.021652 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "0da54287d985460abf463332ab5539ec"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "b15cdccb6d517704b23b2f530d434e81"
server_key_iv: "1227605bb96a6b7dc397d19b0ceee15c"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.022399 11740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "0da54287d985460abf463332ab5539ec"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "b15cdccb6d517704b23b2f530d434e81"
server_key_iv: "1227605bb96a6b7dc397d19b0ceee15c"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.027483 11740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:38:59.030799 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.032217 11740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:59.032369 11740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "0da54287d985460abf463332ab5539ec"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "b15cdccb6d517704b23b2f530d434e81"
server_key_iv: "1227605bb96a6b7dc397d19b0ceee15c"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.032622 11740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:59.038295 11740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:59.039053 11740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:59.039317 11740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:59.056175 11740 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40825
I20260926 03:38:59.056192 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40825 every 8 connection(s)
I20260926 03:38:59.057511 11740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:38:59.061270 11806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:38:59.066330 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11740
I20260926 03:38:59.066524 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:38:59.067198 1910 external_mini_cluster.cc:1428] Setting key 9b76f6e1477b5d2e98110579276964ab
I20260926 03:38:59.073256 11806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec: Bootstrap starting.
I20260926 03:38:59.076989 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:59.069338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.1.221.190:40825)
0926 03:38:59.070577 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:59.070593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:59.070648 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:38:59.072931 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:59.072962 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:59.073528 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:59.073977 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:59.074990 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:59.075003 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:59.075021 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:59.075042 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:59.075118 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:59.075141 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:59.075363 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:59.075514 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 03:38:59.075808 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":377,"threads_started":1}
I20260926 03:38:59.077204 11806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec: Neither blocks nor log segments found. Creating new log.
I20260926 03:38:59.078250 11806 log.cc:824] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec: Log is configured to *not* fsync() on all Append() calls
I20260926 03:38:59.081141 11806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec: No bootstrap required, opened a new log
I20260926 03:38:59.088502 11806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } }
I20260926 03:38:59.088941 11806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:38:59.089061 11806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da54287d985460abf463332ab5539ec, State: Initialized, Role: FOLLOWER
I20260926 03:38:59.089720 11806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } }
I20260926 03:38:59.089958 11806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:38:59.090093 11806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:38:59.090246 11806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:38:59.092396 11806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } }
I20260926 03:38:59.092887 11806 leader_election.cc:302] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da54287d985460abf463332ab5539ec; no voters:
I20260926 03:38:59.093449 11806 leader_election.cc:288] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:38:59.093571 11811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:38:59.094686 11811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [term 1 LEADER]: Becoming Leader. State: Replica: 0da54287d985460abf463332ab5539ec, State: Running, Role: LEADER
I20260926 03:38:59.095233 11811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } }
I20260926 03:38:59.095659 11806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:38:59.097852 11812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da54287d985460abf463332ab5539ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } } }
I20260926 03:38:59.097926 11813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da54287d985460abf463332ab5539ec. Latest consensus state: current_term: 1 leader_uuid: "0da54287d985460abf463332ab5539ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da54287d985460abf463332ab5539ec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40825 } } }
I20260926 03:38:59.098304 11812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:59.098306 11813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec [sys.catalog]: This master's current role is: LEADER
I20260926 03:38:59.100021 11817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:38:59.105744 11817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:38:59.115509 11817 catalog_manager.cc:1347] Generated new cluster ID: fd69be07ebdc4d6493ac53ce65f58732
I20260926 03:38:59.115638 11817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:38:59.127418 11817 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:38:59.127853 11817 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:38:59.137338 11817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da54287d985460abf463332ab5539ec: Generated new TSK 0
I20260926 03:38:59.138116 11817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:38:59.145056 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--builtin_ntp_servers=127.1.221.148:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:38:59.478478 11830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:59.478772 11830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:59.478886 11830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:59.486275 11830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:59.486398 11830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:59.486544 11830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:38:59.496093 11830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:59.497548 11830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:59.499368 11830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:59.512213 11835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.512213 11836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.512213 11838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:59.513021 11830 server_base.cc:1034] running on GCE node
I20260926 03:38:59.513582 11830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:59.514477 11830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:59.515659 11830 hybrid_clock.cc:648] HybridClock initialized: now 1790393939515605 us; error 44 us; skew 500 ppm
I20260926 03:38:59.515957 11830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:59.518065 11830 webserver.cc:460] Webserver started at http://127.1.221.129:45247/ using document root <none> and password file <none>
I20260926 03:38:59.518774 11830 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:59.518875 11830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:59.519177 11830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:59.521986 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "cfdea0c08f0f4036958c2a48f9e36ee1"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "c940c429e7b4d5c1dd378304f4f3be20"
server_key_iv: "2d9ab1cec4ef64d174f91bb4b8da2503"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.522743 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "cfdea0c08f0f4036958c2a48f9e36ee1"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "c940c429e7b4d5c1dd378304f4f3be20"
server_key_iv: "2d9ab1cec4ef64d174f91bb4b8da2503"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.527726 11830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 03:38:59.531032 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.532383 11830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:38:59.532536 11830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "cfdea0c08f0f4036958c2a48f9e36ee1"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "c940c429e7b4d5c1dd378304f4f3be20"
server_key_iv: "2d9ab1cec4ef64d174f91bb4b8da2503"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.532733 11830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:59.538375 11830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:59.539116 11830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:59.539358 11830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:59.540315 11830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:59.541657 11830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:59.541740 11830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.541842 11830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:59.541916 11830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.571743 11830 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:44727
I20260926 03:38:59.571784 11956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:44727 every 8 connection(s)
I20260926 03:38:59.573027 11830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:38:59.579170 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11830
I20260926 03:38:59.579365 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:38:59.580089 1910 external_mini_cluster.cc:1428] Setting key e36aee03cd9effebf71da92eded9940a
I20260926 03:38:59.581097 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:59.575627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:37083 (local address 127.1.221.190:40825)
0926 03:38:59.575836 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:38:59.575843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:38:59.577225 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 03:38:59.578901 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:38:59.578907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:38:59.578974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:38:59.579062 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:38:59.580410 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:38:59.580416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:38:59.580421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:38:59.580441 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:38:59.580471 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:38:59.580483 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:38:59.580597 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:38:59.580769 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:38:59.580836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:38:59.582051 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:38:59.576177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40825 (local address 127.1.221.129:37083)
0926 03:38:59.577066 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 03:38:59.577104 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:38:59.578630 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:38:59.579304 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:38:59.579316 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:38:59.579469 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:38:59.580094 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:38:59.580097 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:38:59.580136 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:38:59.580241 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:38:59.580565 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:38:59.580576 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:38:59.580583 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:38:59.580852 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:38:59.581124 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":251,"threads_started":1}
I20260926 03:38:59.582842 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--builtin_ntp_servers=127.1.221.148:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:38:59.583536 11957 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40825
I20260926 03:38:59.583815 11957 heartbeater.cc:464] Registering TS with master...
I20260926 03:38:59.584365 11957 heartbeater.cc:511] Master 127.1.221.190:40825 requested a full tablet report, sending...
I20260926 03:38:59.586490 11771 ts_manager.cc:194] Registered new tserver with Master: cfdea0c08f0f4036958c2a48f9e36ee1 (127.1.221.129:44727)
W20260926 03:38:59.900692 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:38:59.901024 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:38:59.901165 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:38:59.908571 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:38:59.908680 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:38:59.908835 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:38:59.918090 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:38:59.919497 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:38:59.921301 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:38:59.933928 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.934062 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:38:59.933928 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:38:59.935034 11960 server_base.cc:1034] running on GCE node
I20260926 03:38:59.935643 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:38:59.936671 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:38:59.937887 11960 hybrid_clock.cc:648] HybridClock initialized: now 1790393939937824 us; error 57 us; skew 500 ppm
I20260926 03:38:59.938187 11960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:38:59.940439 11960 webserver.cc:460] Webserver started at http://127.1.221.130:35439/ using document root <none> and password file <none>
I20260926 03:38:59.941139 11960 fs_manager.cc:359] Metadata directory not provided
I20260926 03:38:59.941227 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:38:59.941526 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:38:59.944357 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "2460cf4938a049458b49bd4aa39d6163"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "55e6a226021e9f2c643539425a40278e"
server_key_iv: "8220a9b5e13d2451b169620434fddb43"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.945113 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "2460cf4938a049458b49bd4aa39d6163"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "55e6a226021e9f2c643539425a40278e"
server_key_iv: "8220a9b5e13d2451b169620434fddb43"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.950058 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:38:59.953409 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.954771 11960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 03:38:59.954952 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "2460cf4938a049458b49bd4aa39d6163"
format_stamp: "Formatted at 2026-09-26 03:38:59 on dist-test-slave-lqrk"
server_key: "55e6a226021e9f2c643539425a40278e"
server_key_iv: "8220a9b5e13d2451b169620434fddb43"
server_key_version: "encryptionkey@0"
I20260926 03:38:59.955236 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:38:59.963100 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:38:59.964145 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:38:59.964422 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:38:59.965546 11960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:38:59.967509 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:38:59.967625 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:38:59.967738 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:38:59.967811 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:00.000994 11960 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:33505
I20260926 03:39:00.001034 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:33505 every 8 connection(s)
I20260926 03:39:00.002931 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:39:00.004799 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 11960
I20260926 03:39:00.004957 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:39:00.005406 1910 external_mini_cluster.cc:1428] Setting key 7fcc880c2834b5064e1f1368706a0da4
I20260926 03:39:00.008315 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--builtin_ntp_servers=127.1.221.148:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:39:00.011629 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.005603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56411 (local address 127.1.221.190:40825)
0926 03:39:00.005812 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.005818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.007627 (+ 1809us) server_negotiation.cc:407] Connection header received
0926 03:39:00.009395 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.009400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.009463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.009559 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.010869 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:00.010876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:00.010881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:39:00.010898 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:00.010930 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:39:00.010942 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:00.011059 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:00.011242 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 03:39:00.011320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:39:00.012557 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.006248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40825 (local address 127.1.221.130:56411)
0926 03:39:00.007466 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:00.007516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:00.009118 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:00.009795 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:00.009803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:00.009901 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:00.010549 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:39:00.010552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:00.010604 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:39:00.010695 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:00.011025 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:00.011037 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:00.011061 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:39:00.011313 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 03:39:00.011617 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
I20260926 03:39:00.013871 12087 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40825
I20260926 03:39:00.014204 12087 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:00.014813 12087 heartbeater.cc:511] Master 127.1.221.190:40825 requested a full tablet report, sending...
I20260926 03:39:00.016294 11771 ts_manager.cc:194] Registered new tserver with Master: 2460cf4938a049458b49bd4aa39d6163 (127.1.221.130:33505)
W20260926 03:39:00.328924 12090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:00.329212 12090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:00.329344 12090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:00.336773 12090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:00.336898 12090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:00.337052 12090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:39:00.346495 12090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:00.347894 12090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:00.349704 12090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:00.362695 12095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:00.362654 12096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:00.362731 12098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:00.364140 12090 server_base.cc:1034] running on GCE node
I20260926 03:39:00.364895 12090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:00.365972 12090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:00.367182 12090 hybrid_clock.cc:648] HybridClock initialized: now 1790393940367110 us; error 67 us; skew 500 ppm
I20260926 03:39:00.367550 12090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:39:00.369859 12090 webserver.cc:460] Webserver started at http://127.1.221.131:43999/ using document root <none> and password file <none>
I20260926 03:39:00.370565 12090 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:00.370676 12090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:00.370996 12090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:39:00.373871 12090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "49e38c9049ea49a9b64dc332b24475c9"
format_stamp: "Formatted at 2026-09-26 03:39:00 on dist-test-slave-lqrk"
server_key: "d24b6f71becdccc12d52dec672a235c6"
server_key_iv: "ecd8e92f4f75310600c17963f994d0a0"
server_key_version: "encryptionkey@0"
I20260926 03:39:00.374650 12090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "49e38c9049ea49a9b64dc332b24475c9"
format_stamp: "Formatted at 2026-09-26 03:39:00 on dist-test-slave-lqrk"
server_key: "d24b6f71becdccc12d52dec672a235c6"
server_key_iv: "ecd8e92f4f75310600c17963f994d0a0"
server_key_version: "encryptionkey@0"
I20260926 03:39:00.379750 12090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:39:00.383181 12104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:00.384696 12090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:39:00.384857 12090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "49e38c9049ea49a9b64dc332b24475c9"
format_stamp: "Formatted at 2026-09-26 03:39:00 on dist-test-slave-lqrk"
server_key: "d24b6f71becdccc12d52dec672a235c6"
server_key_iv: "ecd8e92f4f75310600c17963f994d0a0"
server_key_version: "encryptionkey@0"
I20260926 03:39:00.385066 12090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:00.390923 12090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:00.391778 12090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:00.392024 12090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:00.392943 12090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:00.394335 12090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:00.394428 12090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:00.394534 12090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:00.394610 12090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:00.425287 12090 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:38971
I20260926 03:39:00.425365 12216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:38971 every 8 connection(s)
I20260926 03:39:00.426914 12090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:39:00.433408 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 12090
I20260926 03:39:00.433593 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:39:00.434191 1910 external_mini_cluster.cc:1428] Setting key f861455b94e7e6eb0778f4ec58881fec
I20260926 03:39:00.441838 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.437337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59535 (local address 127.1.221.190:40825)
0926 03:39:00.437511 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.437517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.437811 (+ 294us) server_negotiation.cc:407] Connection header received
0926 03:39:00.440048 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.440052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.440107 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.440187 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.441308 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:00.441312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:00.441315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:39:00.441330 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:00.441360 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:39:00.441371 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:00.441456 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:00.441584 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:39:00.441639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:39:00.442759 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.436443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40825 (local address 127.1.221.131:59535)
0926 03:39:00.437650 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:00.437709 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:00.439803 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:00.440373 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:00.440381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:00.440476 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:00.441034 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:39:00.441036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:00.441072 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:39:00.441166 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:00.441441 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:00.441446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:00.441451 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:39:00.441646 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 03:39:00.441845 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":336,"threads_started":1}
I20260926 03:39:00.443953 12217 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40825
I20260926 03:39:00.444209 12217 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:00.444741 12217 heartbeater.cc:511] Master 127.1.221.190:40825 requested a full tablet report, sending...
I20260926 03:39:00.445927 11771 ts_manager.cc:194] Registered new tserver with Master: 49e38c9049ea49a9b64dc332b24475c9 (127.1.221.131:38971)
I20260926 03:39:00.450225 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:39:00.455999 11809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:39:00.454503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.1.221.190:40825)
0926 03:39:00.454654 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.454659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.455135 (+ 476us) server_negotiation.cc:407] Connection header received
0926 03:39:00.455260 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.455263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.455314 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.455427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.455761 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52316: BlockingRecv error: recv got EOF from 127.0.0.1:52316 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:39:00.456314 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11830
I20260926 03:39:00.460006 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.457297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.1.221.129:44727)
0926 03:39:00.457805 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.457820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.457848 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:39:00.457922 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.457928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.458139 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.458284 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.459257 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:00.459264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:00.459276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:39:00.459288 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:00.459329 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:39:00.459343 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:00.459529 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:00.459585 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 03:39:00.459684 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":229,"threads_started":1}
I20260926 03:39:00.461314 11952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:00.541935 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11830
I20260926 03:39:00.555204 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11960
I20260926 03:39:00.558657 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.556224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.1.221.130:33505)
0926 03:39:00.556708 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.556715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.556753 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:39:00.556815 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.556822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.557041 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.557165 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.558025 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:00.558032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:00.558043 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:39:00.558060 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:00.558100 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:39:00.558114 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:00.558215 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:00.558284 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 03:39:00.558371 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260926 03:39:00.560014 12082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:00.639703 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11960
I20260926 03:39:00.652361 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 12090
I20260926 03:39:00.656140 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:00.653351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.1.221.131:38971)
0926 03:39:00.653958 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:00.653965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:00.653996 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:39:00.654057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:00.654063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:00.654305 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:00.654433 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:00.655479 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:00.655490 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:00.655503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:39:00.655522 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:00.655576 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:39:00.655592 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:00.655722 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:00.655775 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 03:39:00.655860 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":308,"threads_started":1}
I20260926 03:39:00.657465 12212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:00.737609 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 12090
I20260926 03:39:00.750504 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 11740
I20260926 03:39:00.751865 11801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:00.851824 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 11740
2026-09-26T03:39:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2251 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:39:00 dist-test-slave-lqrk krb5kdc[12235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:39:00 dist-test-slave-lqrk krb5kdc[12235](info): set up 2 sockets
Sep 26 03:39:00 dist-test-slave-lqrk krb5kdc[12235](info): commencing operation
krb5kdc: starting...
W20260926 03:39:01.868268 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:39:02 dist-test-slave-lqrk krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:39:02.090444 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34145
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:39:02.408087 12248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:02.408387 12248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:02.408519 12248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:02.415762 12248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:39:02.415874 12248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:02.415944 12248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:02.416023 12248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:39:02.416081 12248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:39:02.427579 12248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:02.429417 12248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:02.430773 12248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:02.442174 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:02.442245 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:02.442766 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:02.443010 12248 server_base.cc:1034] running on GCE node
I20260926 03:39:02.443878 12248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:02.445324 12248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:02.446518 12248 hybrid_clock.cc:648] HybridClock initialized: now 1790393942446461 us; error 47 us; skew 500 ppm
I20260926 03:39:02.446851 12248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:02 dist-test-slave-lqrk krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:02.453244 12248 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:39:02.454728 12248 webserver.cc:460] Webserver started at http://127.1.221.190:45321/ using document root <none> and password file <none>
I20260926 03:39:02.455488 12248 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:02.455612 12248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:02.455904 12248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:39:02.458942 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9"
format_stamp: "Formatted at 2026-09-26 03:39:02 on dist-test-slave-lqrk"
server_key: "102591712eef949be3bb1c12a7b13155"
server_key_iv: "1e8e93e8149edc7ad244c704e0d429e1"
server_key_version: "encryptionkey@0"
I20260926 03:39:02.459740 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9"
format_stamp: "Formatted at 2026-09-26 03:39:02 on dist-test-slave-lqrk"
server_key: "102591712eef949be3bb1c12a7b13155"
server_key_iv: "1e8e93e8149edc7ad244c704e0d429e1"
server_key_version: "encryptionkey@0"
I20260926 03:39:02.464819 12248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:39:02.468076 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:02.469409 12248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:39:02.469554 12248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9"
format_stamp: "Formatted at 2026-09-26 03:39:02 on dist-test-slave-lqrk"
server_key: "102591712eef949be3bb1c12a7b13155"
server_key_iv: "1e8e93e8149edc7ad244c704e0d429e1"
server_key_version: "encryptionkey@0"
I20260926 03:39:02.469729 12248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:02.484391 12248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:02.488220 12248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:02.488474 12248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:02.505636 12248 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34145
I20260926 03:39:02.505658 12314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34145 every 8 connection(s)
I20260926 03:39:02.506963 12248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:39:02.510661 12315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:02.511193 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 12248
I20260926 03:39:02.511443 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:39:02.512228 1910 external_mini_cluster.cc:1428] Setting key 3a0fbb5b04c5beb1c99136388d9b1b7f
I20260926 03:39:02.524703 12315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: Bootstrap starting.
Sep 26 03:39:02 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:02.527849 12315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:02.529199 12315 log.cc:824] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:02.532470 12315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: No bootstrap required, opened a new log
I20260926 03:39:02.535188 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:02.514218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.1.221.190:34145)
0926 03:39:02.514955 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:02.514968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:02.515009 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:39:02.516910 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:02.516934 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:02.517344 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:02.517682 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:02.519374 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:02.520674 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:02.522420 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:02.523489 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:02.523663 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:02.527290 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:02.527301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:02.527320 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:02.527368 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:02.530875 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:02.531584 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:02.531592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:02.531600 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:02.531807 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:02.532143 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:02.532147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:02.532151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:02.532819 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:02.533140 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:02.533416 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:39:02.533832 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":332,"threads_started":1}
I20260926 03:39:02.541236 12315 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } }
I20260926 03:39:02.541579 12315 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:02.541652 12315 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cfe5fbb3a9440c8ac00932d689e6e9, State: Initialized, Role: FOLLOWER
I20260926 03:39:02.542140 12315 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } }
I20260926 03:39:02.542336 12315 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:39:02.542454 12315 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:39:02.542598 12315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:02.544962 12315 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } }
I20260926 03:39:02.545418 12315 leader_election.cc:302] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cfe5fbb3a9440c8ac00932d689e6e9; no voters:
I20260926 03:39:02.545969 12315 leader_election.cc:288] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:39:02.546082 12320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:39:02.546653 12320 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [term 1 LEADER]: Becoming Leader. State: Replica: e3cfe5fbb3a9440c8ac00932d689e6e9, State: Running, Role: LEADER
I20260926 03:39:02.547158 12320 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } }
I20260926 03:39:02.547889 12315 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:39:02.551054 12321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } } }
I20260926 03:39:02.551282 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cfe5fbb3a9440c8ac00932d689e6e9. Latest consensus state: current_term: 1 leader_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cfe5fbb3a9440c8ac00932d689e6e9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34145 } } }
I20260926 03:39:02.551535 12321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:39:02.551601 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:39:02.552371 12328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:39:02.557353 12328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:39:02.567229 12328 catalog_manager.cc:1347] Generated new cluster ID: f381e2f1abd74f52bddc97905eb7fc54
I20260926 03:39:02.567348 12328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:39:02.579835 12328 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:39:02.581148 12328 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:39:02.592971 12328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: Generated new TSK 0
I20260926 03:39:02.593835 12328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:39:02.662966 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:39:02.989318 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:02.989604 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:02.989697 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:02.996974 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:02.997090 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:02.997203 12341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:39:03.006866 12341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:03.008172 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:03.009429 12341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:03.021778 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.021818 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.021839 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:03.022763 12341 server_base.cc:1034] running on GCE node
I20260926 03:39:03.023301 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:03.024233 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:03.025405 12341 hybrid_clock.cc:648] HybridClock initialized: now 1790393943025355 us; error 41 us; skew 500 ppm
I20260926 03:39:03.025755 12341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:03 dist-test-slave-lqrk krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:03.028743 12341 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:39:03.030083 12341 webserver.cc:460] Webserver started at http://127.1.221.129:39799/ using document root <none> and password file <none>
I20260926 03:39:03.030745 12341 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:03.030856 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:03.031168 12341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:39:03.034211 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "b0194615c18947e2911e9d93b72afe0f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "e6f1192ea5ad6b96c13a40dc405e3eaa"
server_key_iv: "1474ff7475f95bbdf507e28d082d4f85"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.035084 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "b0194615c18947e2911e9d93b72afe0f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "e6f1192ea5ad6b96c13a40dc405e3eaa"
server_key_iv: "1474ff7475f95bbdf507e28d082d4f85"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.040126 12341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:39:03.043413 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.044768 12341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:39:03.044929 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "b0194615c18947e2911e9d93b72afe0f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "e6f1192ea5ad6b96c13a40dc405e3eaa"
server_key_iv: "1474ff7475f95bbdf507e28d082d4f85"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.045110 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:03.058617 12341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:03.062544 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:03.062772 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:03.063629 12341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:03.064867 12341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:03.064935 12341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.065016 12341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:03.065066 12341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.093694 12341 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:34999
I20260926 03:39:03.093730 12468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:34999 every 8 connection(s)
I20260926 03:39:03.094899 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:39:03.105005 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 12341
I20260926 03:39:03.105206 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:39:03.105902 1910 external_mini_cluster.cc:1428] Setting key ccdb33048f8741bceb106af66a741480
Sep 26 03:39:03 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:03.113948 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:03.097337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:46929 (local address 127.1.221.190:34145)
0926 03:39:03.097736 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:03.097740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:03.098659 (+ 919us) server_negotiation.cc:407] Connection header received
0926 03:39:03.100402 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:03.100408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:03.100488 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:03.100581 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:03.101997 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.102702 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.104227 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.104614 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.104728 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:03.108093 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:03.108099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:03.108103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:03.108140 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:03.110937 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:03.111735 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:03.111740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:03.111743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:03.111870 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:03.112270 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:03.112274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:03.112277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:03.112574 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:03.112696 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:03.113403 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 03:39:03.113617 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":213,"threads_started":1}
I20260926 03:39:03.114815 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:03.097815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34145 (local address 127.1.221.129:46929)
0926 03:39:03.098489 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:03.098524 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:03.100141 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:03.100816 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:03.100825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:03.101309 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:03.101842 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:03.101858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.102845 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.102849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:03.104093 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:03.104104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.104820 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.104827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:03.105049 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:03.105706 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:03.105727 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:03.107888 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:03.111086 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:03.111095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:03.111112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:03.111610 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:03.111958 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:03.111962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:03.111965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:03.112173 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:03.112677 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:03.112684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:03.113040 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 03:39:03.113281 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 03:39:03.113732 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
I20260926 03:39:03.116291 12469 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34145
I20260926 03:39:03.116632 12469 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:03.117340 12469 heartbeater.cc:511] Master 127.1.221.190:34145 requested a full tablet report, sending...
I20260926 03:39:03.119333 12280 ts_manager.cc:194] Registered new tserver with Master: b0194615c18947e2911e9d93b72afe0f (127.1.221.129:34999)
I20260926 03:39:03.120766 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:46929
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:39:03.154496 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:39:03.470492 12475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:03.470783 12475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:03.470925 12475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:03.478226 12475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:03.478353 12475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:03.478509 12475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:39:03.488219 12475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:03.489529 12475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:03.490803 12475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:03.503473 12483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.503496 12481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.503531 12480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:03.504237 12475 server_base.cc:1034] running on GCE node
I20260926 03:39:03.504802 12475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:03.505739 12475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:03.506978 12475 hybrid_clock.cc:648] HybridClock initialized: now 1790393943506930 us; error 43 us; skew 500 ppm
I20260926 03:39:03.507282 12475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:03 dist-test-slave-lqrk krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:03.510208 12475 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:39:03.511650 12475 webserver.cc:460] Webserver started at http://127.1.221.130:44543/ using document root <none> and password file <none>
I20260926 03:39:03.512275 12475 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:03.512382 12475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:03.512655 12475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:39:03.515534 12475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "fe9cbdc4ebe040fba42d63089e6124e6"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "897c25dec7fedc8a0b6f96259d68a427"
server_key_iv: "9c47259c19797b967e83b6ac57908fe9"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.516290 12475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "fe9cbdc4ebe040fba42d63089e6124e6"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "897c25dec7fedc8a0b6f96259d68a427"
server_key_iv: "9c47259c19797b967e83b6ac57908fe9"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.521286 12475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:39:03.524699 12490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.526022 12475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:39:03.526182 12475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "fe9cbdc4ebe040fba42d63089e6124e6"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "897c25dec7fedc8a0b6f96259d68a427"
server_key_iv: "9c47259c19797b967e83b6ac57908fe9"
server_key_version: "encryptionkey@0"
I20260926 03:39:03.526367 12475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:03.549926 12475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:03.554183 12475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:03.554450 12475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:03.555469 12475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:03.556906 12475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:03.556981 12475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.557097 12475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:03.557155 12475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:03.587044 12475 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:37301
I20260926 03:39:03.587070 12602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:37301 every 8 connection(s)
I20260926 03:39:03.588343 12475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:39:03.594522 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 12475
I20260926 03:39:03.594704 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:39:03.595111 1910 external_mini_cluster.cc:1428] Setting key a3560ff4edd4f6a02145bc0fb7428e0d
Sep 26 03:39:03 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:03.607688 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:03.590799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34541 (local address 127.1.221.190:34145)
0926 03:39:03.590971 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:03.590976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:03.592166 (+ 1190us) server_negotiation.cc:407] Connection header received
0926 03:39:03.593916 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:03.593921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:03.594006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:03.594101 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:03.595707 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.596572 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.598172 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.598582 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.598691 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:03.602127 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:03.602133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:03.602136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:03.602168 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:03.604570 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:03.605376 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:03.605381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:03.605384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:03.605519 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:03.605950 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:03.605953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:03.605955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:03.606215 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:03.606324 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:03.607189 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 03:39:03.607374 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:39:03.608852 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:03.591273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34145 (local address 127.1.221.130:34541)
0926 03:39:03.592008 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:03.592047 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:03.593654 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:03.594345 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:03.594353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:03.594925 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:03.595551 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:03.595571 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.596704 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.596708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:03.598026 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:03.598037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:03.598788 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:03.598795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:03.599005 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:03.599647 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:03.599670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:03.601901 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:03.604702 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:03.604712 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:03.604730 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:03.605249 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:03.605617 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:03.605622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:03.605625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:03.605850 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:03.606322 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:03.606330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:03.606755 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 03:39:03.607026 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 03:39:03.607559 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":257,"threads_started":1}
I20260926 03:39:03.610248 12603 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34145
I20260926 03:39:03.610534 12603 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:03.611212 12603 heartbeater.cc:511] Master 127.1.221.190:34145 requested a full tablet report, sending...
I20260926 03:39:03.612619 12280 ts_manager.cc:194] Registered new tserver with Master: fe9cbdc4ebe040fba42d63089e6124e6 (127.1.221.130:37301)
I20260926 03:39:03.613404 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:34541
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:39:03.646616 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:39:03.958245 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:03.958513 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:03.958637 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:03.965648 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:03.965756 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:03.965901 12608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:39:03.975350 12608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:03.976629 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:03.977860 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:03.990449 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.990482 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:03.990563 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:03.991271 12608 server_base.cc:1034] running on GCE node
I20260926 03:39:03.991919 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:03.992875 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:03.994171 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790393943994115 us; error 37 us; skew 500 ppm
I20260926 03:39:03.994480 12608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:03 dist-test-slave-lqrk krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:03.997511 12608 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:39:03.998965 12608 webserver.cc:460] Webserver started at http://127.1.221.131:39015/ using document root <none> and password file <none>
I20260926 03:39:03.999719 12608 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:03.999840 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:04.000136 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:39:04.003121 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "cc9f3fa0a530476d80ff9d59b894ed8f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "f237ba60d9c94e4aa827beecd9999022"
server_key_iv: "2eccfb535dd66d636a5d69ea7799e383"
server_key_version: "encryptionkey@0"
I20260926 03:39:04.004036 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "cc9f3fa0a530476d80ff9d59b894ed8f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "f237ba60d9c94e4aa827beecd9999022"
server_key_iv: "2eccfb535dd66d636a5d69ea7799e383"
server_key_version: "encryptionkey@0"
I20260926 03:39:04.009399 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:39:04.012916 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:04.014294 12608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:39:04.014461 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "cc9f3fa0a530476d80ff9d59b894ed8f"
format_stamp: "Formatted at 2026-09-26 03:39:03 on dist-test-slave-lqrk"
server_key: "f237ba60d9c94e4aa827beecd9999022"
server_key_iv: "2eccfb535dd66d636a5d69ea7799e383"
server_key_version: "encryptionkey@0"
I20260926 03:39:04.014650 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:04.037660 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:04.041821 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:04.042085 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:04.043015 12608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:04.044322 12608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:04.044397 12608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:04.044504 12608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:04.044555 12608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:04.073953 12608 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:42459
I20260926 03:39:04.073987 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:42459 every 8 connection(s)
I20260926 03:39:04.075277 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:39:04.077524 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 12608
I20260926 03:39:04.077739 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:39:04.078254 1910 external_mini_cluster.cc:1428] Setting key d81d904af3e36460820d94c6f3b3ba08
Sep 26 03:39:04 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:04.095054 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.078877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:56977 (local address 127.1.221.190:34145)
0926 03:39:04.079050 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.079056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.079513 (+ 457us) server_negotiation.cc:407] Connection header received
0926 03:39:04.081734 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.081739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.081824 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.081907 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:04.083352 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.084132 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.085634 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.086041 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.086149 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.089479 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:04.089485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:04.089488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:04.089522 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:04.091912 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.092664 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.092669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.092671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.092796 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.093314 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.093319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.093323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.093668 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:04.093757 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.094504 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.094715 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:39:04.096053 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.078438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34145 (local address 127.1.221.131:56977)
0926 03:39:04.079343 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.079413 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.081490 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.082144 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.082153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.082653 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:04.083204 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.083220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.084276 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.084280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.085505 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.085517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.086250 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.086256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.086479 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.087086 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:04.087108 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:04.089251 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:04.092039 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:04.092049 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:04.092067 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:04.092559 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:04.092894 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:04.092899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:04.092903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:04.093188 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:04.093843 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:04.093848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:04.094157 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.094392 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.094903 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":366,"threads_started":1}
I20260926 03:39:04.097373 12736 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34145
I20260926 03:39:04.097680 12736 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:04.098364 12736 heartbeater.cc:511] Master 127.1.221.190:34145 requested a full tablet report, sending...
I20260926 03:39:04.099705 12280 ts_manager.cc:194] Registered new tserver with Master: cc9f3fa0a530476d80ff9d59b894ed8f (127.1.221.131:42459)
I20260926 03:39:04.100440 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:56977
I20260926 03:39:04.105681 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:39:04.123623 12469 heartbeater.cc:503] Master 127.1.221.190:34145 was elected leader, sending a full tablet report...
I20260926 03:39:04.124291 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.110023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.1.221.190:34145)
0926 03:39:04.110182 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.110187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.110662 (+ 475us) server_negotiation.cc:407] Connection header received
0926 03:39:04.110798 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.110802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.110861 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.110938 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:04.112295 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.113075 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.114716 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.115359 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.115526 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.118464 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:04.118471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:04.118476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:04.118521 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:04.121834 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.122497 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.122500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.122503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.122625 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.122918 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.122921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.122923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.123230 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:04.123456 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.123755 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.123961 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:39:04.128396 12279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:39:04.135169 12279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:39:04.166249 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.147717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.1.221.129:34999)
0926 03:39:04.157499 (+ 9782us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.157510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.158797 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 03:39:04.158909 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.158917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.159211 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.159437 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:04.160402 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.161893 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.164200 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.165118 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.165238 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.165475 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.165765 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.165926 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9579,"thread_start_us":9386,"threads_started":1}
I20260926 03:39:04.166253 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.146732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:34999 (local address 127.0.0.1:41828)
0926 03:39:04.158457 (+ 11725us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.158476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.158650 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.159772 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.159781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.159815 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:04.160187 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.160197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.162048 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.162053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.164058 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.164070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.165335 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.165341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.165550 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.165588 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.165687 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.165890 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11591,"thread_start_us":7883,"threads_started":1}
I20260926 03:39:04.168344 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.147174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42459 (local address 127.0.0.1:43004)
0926 03:39:04.158891 (+ 11717us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.158908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.159003 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.161037 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.161042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.161066 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:04.161500 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.161512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.163417 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.163422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.165841 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.165855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.167557 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.167565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.167718 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.167748 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.167842 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.167959 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11579,"mutex_wait_us":110,"thread_start_us":11010,"threads_started":1}
I20260926 03:39:04.168392 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.147626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.1.221.131:42459)
0926 03:39:04.160513 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.160521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.160553 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:39:04.160643 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.160650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.160858 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.161037 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:04.161702 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.163254 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.166007 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.167350 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.167513 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.167721 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.167884 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.168043 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12695,"thread_start_us":12526,"threads_started":1}
I20260926 03:39:04.169360 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.146596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37301 (local address 127.0.0.1:35126)
0926 03:39:04.155017 (+ 8421us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.155059 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.155249 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.161338 (+ 6089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.161343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.161364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:04.161734 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.161742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.163374 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.163406 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.166880 (+ 3474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.166894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.168833 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.168837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.168942 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.168962 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.169038 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.169119 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8144,"mutex_wait_us":134,"thread_start_us":12024,"threads_started":1}
I20260926 03:39:04.169565 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.147111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.1.221.130:37301)
0926 03:39:04.160765 (+ 13654us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.160774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.160804 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:39:04.160892 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.160899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.161149 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.161322 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:04.161891 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.163246 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.167066 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.168444 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.168568 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.168732 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.169111 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.169266 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13499,"thread_start_us":13384,"threads_started":1}
I20260926 03:39:04.170313 12404 tablet_service.cc:1467] Processing CreateTablet for tablet b09160c57f474dec975e3aa2980213d2 (DEFAULT_TABLE table=test-table [id=38d32517cbe047de85fc23436652b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:04.171561 12404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09160c57f474dec975e3aa2980213d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:04.172425 12538 tablet_service.cc:1467] Processing CreateTablet for tablet b09160c57f474dec975e3aa2980213d2 (DEFAULT_TABLE table=test-table [id=38d32517cbe047de85fc23436652b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:04.173437 12538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09160c57f474dec975e3aa2980213d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:04.177068 12671 tablet_service.cc:1467] Processing CreateTablet for tablet b09160c57f474dec975e3aa2980213d2 (DEFAULT_TABLE table=test-table [id=38d32517cbe047de85fc23436652b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:04.178395 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09160c57f474dec975e3aa2980213d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:04.183835 12754 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Bootstrap starting.
I20260926 03:39:04.186347 12755 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Bootstrap starting.
I20260926 03:39:04.188078 12754 tablet_bootstrap.cc:654] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:04.189306 12754 log.cc:824] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:04.189677 12755 tablet_bootstrap.cc:654] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:04.190712 12755 log.cc:824] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:04.192350 12754 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: No bootstrap required, opened a new log
I20260926 03:39:04.192629 12754 ts_tablet_manager.cc:1397] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:39:04.193789 12755 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: No bootstrap required, opened a new log
I20260926 03:39:04.194051 12755 ts_tablet_manager.cc:1397] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:39:04.195003 12757 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Bootstrap starting.
I20260926 03:39:04.197678 12757 tablet_bootstrap.cc:654] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:04.198619 12757 log.cc:824] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:04.201006 12757 tablet_bootstrap.cc:492] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: No bootstrap required, opened a new log
I20260926 03:39:04.201242 12757 ts_tablet_manager.cc:1397] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 03:39:04.202967 12754 raft_consensus.cc:348] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.203563 12754 raft_consensus.cc:374] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:04.203704 12754 raft_consensus.cc:729] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9cbdc4ebe040fba42d63089e6124e6, State: Initialized, Role: FOLLOWER
I20260926 03:39:04.204402 12754 consensus_queue.cc:260] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.204794 12755 raft_consensus.cc:348] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.205551 12603 heartbeater.cc:503] Master 127.1.221.190:34145 was elected leader, sending a full tablet report...
I20260926 03:39:04.206310 12754 ts_tablet_manager.cc:1428] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260926 03:39:04.205307 12755 raft_consensus.cc:374] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:04.206566 12755 raft_consensus.cc:729] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0194615c18947e2911e9d93b72afe0f, State: Initialized, Role: FOLLOWER
I20260926 03:39:04.207253 12755 consensus_queue.cc:260] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.210063 12757 raft_consensus.cc:348] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.210500 12755 ts_tablet_manager.cc:1428] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260926 03:39:04.210580 12757 raft_consensus.cc:374] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:04.210711 12757 raft_consensus.cc:729] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9f3fa0a530476d80ff9d59b894ed8f, State: Initialized, Role: FOLLOWER
I20260926 03:39:04.211350 12757 consensus_queue.cc:260] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.212494 12736 heartbeater.cc:503] Master 127.1.221.190:34145 was elected leader, sending a full tablet report...
I20260926 03:39:04.213099 12757 ts_tablet_manager.cc:1428] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260926 03:39:04.326859 12737 tablet.cc:2369] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:39:04.340205 12604 tablet.cc:2369] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:39:04.347081 12470 tablet.cc:2369] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:39:04.508172 12761 raft_consensus.cc:482] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:39:04.508499 12761 raft_consensus.cc:504] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.510221 12761 leader_election.cc:288] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe9cbdc4ebe040fba42d63089e6124e6 (127.1.221.130:37301), cc9f3fa0a530476d80ff9d59b894ed8f (127.1.221.131:42459)
I20260926 03:39:04.517696 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.511216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:52153 (local address 127.1.221.131:42459)
0926 03:39:04.511439 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.511445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.511796 (+ 351us) server_negotiation.cc:407] Connection header received
0926 03:39:04.511944 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.511949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.512024 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.512107 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:04.512957 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.513804 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.515555 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.516689 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.516847 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.516932 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.517273 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.517432 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:39:04.517863 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.510772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36309 (local address 127.1.221.130:37301)
0926 03:39:04.510984 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.510990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.511489 (+ 499us) server_negotiation.cc:407] Connection header received
0926 03:39:04.511797 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.511803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.511882 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.511964 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:04.512485 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.513376 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.515856 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.517000 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.517129 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.517196 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.517483 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.517607 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:39:04.518024 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.510590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37301 (local address 127.1.221.129:36309)
0926 03:39:04.511338 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.511364 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.511617 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.511995 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.512000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.512022 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:04.512348 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.512358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.513504 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.513510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.515691 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.515705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.517203 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.517210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.517344 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.517367 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.517443 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.517626 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":414,"threads_started":1}
I20260926 03:39:04.518381 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.510737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42459 (local address 127.1.221.129:52153)
0926 03:39:04.511705 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:04.511720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:04.511823 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:04.512366 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:04.512371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:04.512413 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:04.512740 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.512750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.513927 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.513932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.515420 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:04.515435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.516888 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.516893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:04.517067 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.517104 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:39:04.517891 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 03:39:04.518033 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":661,"threads_started":1}
I20260926 03:39:04.519272 12558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09160c57f474dec975e3aa2980213d2" candidate_uuid: "b0194615c18947e2911e9d93b72afe0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" is_pre_election: true
I20260926 03:39:04.519857 12558 raft_consensus.cc:2454] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0194615c18947e2911e9d93b72afe0f in term 0.
I20260926 03:39:04.520516 12357 leader_election.cc:302] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0194615c18947e2911e9d93b72afe0f, fe9cbdc4ebe040fba42d63089e6124e6; no voters:
I20260926 03:39:04.520654 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09160c57f474dec975e3aa2980213d2" candidate_uuid: "b0194615c18947e2911e9d93b72afe0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" is_pre_election: true
I20260926 03:39:04.520953 12761 raft_consensus.cc:2781] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:39:04.521095 12761 raft_consensus.cc:482] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:39:04.521117 12691 raft_consensus.cc:2454] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0194615c18947e2911e9d93b72afe0f in term 0.
I20260926 03:39:04.521173 12761 raft_consensus.cc:3037] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:04.523448 12761 raft_consensus.cc:504] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.524045 12761 leader_election.cc:288] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [CANDIDATE]: Term 1 election: Requested vote from peers fe9cbdc4ebe040fba42d63089e6124e6 (127.1.221.130:37301), cc9f3fa0a530476d80ff9d59b894ed8f (127.1.221.131:42459)
I20260926 03:39:04.524567 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09160c57f474dec975e3aa2980213d2" candidate_uuid: "b0194615c18947e2911e9d93b72afe0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f"
I20260926 03:39:04.524574 12558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09160c57f474dec975e3aa2980213d2" candidate_uuid: "b0194615c18947e2911e9d93b72afe0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9cbdc4ebe040fba42d63089e6124e6"
I20260926 03:39:04.524737 12691 raft_consensus.cc:3037] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:04.524767 12558 raft_consensus.cc:3037] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:04.526871 12691 raft_consensus.cc:2454] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0194615c18947e2911e9d93b72afe0f in term 1.
I20260926 03:39:04.526906 12558 raft_consensus.cc:2454] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0194615c18947e2911e9d93b72afe0f in term 1.
I20260926 03:39:04.527307 12358 leader_election.cc:302] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0194615c18947e2911e9d93b72afe0f, cc9f3fa0a530476d80ff9d59b894ed8f; no voters:
I20260926 03:39:04.527702 12761 raft_consensus.cc:2781] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:39:04.528137 12761 raft_consensus.cc:686] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 1 LEADER]: Becoming Leader. State: Replica: b0194615c18947e2911e9d93b72afe0f, State: Running, Role: LEADER
I20260926 03:39:04.528614 12761 consensus_queue.cc:237] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } }
I20260926 03:39:04.533499 12279 catalog_manager.cc:5495] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f reported cstate change: term changed from 0 to 1, leader changed from <none> to b0194615c18947e2911e9d93b72afe0f (127.1.221.129). New cstate: current_term: 1 leader_uuid: "b0194615c18947e2911e9d93b72afe0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0194615c18947e2911e9d93b72afe0f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 } health_report { overall_health: UNKNOWN } } }
I20260926 03:39:04.628973 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.623692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.1.221.129:34999)
0926 03:39:04.623906 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.623912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.623931 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:39:04.624036 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.624039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.624131 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.624261 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:39:04.624649 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.625398 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.626765 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.627182 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.627296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.627829 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:39:04.628371 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:39:04.628506 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.628630 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.628717 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:39:04.637302 12558 raft_consensus.cc:1261] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 1 FOLLOWER]: Refusing update from remote peer b0194615c18947e2911e9d93b72afe0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:39:04.637307 12691 raft_consensus.cc:1261] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 1 FOLLOWER]: Refusing update from remote peer b0194615c18947e2911e9d93b72afe0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:39:04.638216 12765 consensus_queue.cc:1035] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9f3fa0a530476d80ff9d59b894ed8f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:39:04.638535 12761 consensus_queue.cc:1035] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe9cbdc4ebe040fba42d63089e6124e6" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:39:04.650439 12770 mvcc.cc:205] Tried to move back new op lower bound from 7333453597224681472 to 7333453596793184256. Current Snapshot: MvccSnapshot[applied={T|T < 7333453597224681472}]
I20260926 03:39:04.663151 12278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:39:04.663558 12278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:39:04.667312 12278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3cfe5fbb3a9440c8ac00932d689e6e9: Sending DeleteTablet for 3 replicas of tablet b09160c57f474dec975e3aa2980213d2
I20260926 03:39:04.668541 12404 tablet_service.cc:1514] Processing DeleteTablet for tablet b09160c57f474dec975e3aa2980213d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:04 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41828
I20260926 03:39:04.668560 12538 tablet_service.cc:1514] Processing DeleteTablet for tablet b09160c57f474dec975e3aa2980213d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:04 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:35126
I20260926 03:39:04.668623 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet b09160c57f474dec975e3aa2980213d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:04 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43004
I20260926 03:39:04.676208 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 12341
I20260926 03:39:04.678881 12779 tablet_replica.cc:330] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: stopping tablet replica
I20260926 03:39:04.680111 12779 raft_consensus.cc:2229] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:39:04.681356 12779 raft_consensus.cc:2258] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:04.684864 12779 ts_tablet_manager.cc:1905] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:39:04.691865 12778 tablet_replica.cc:330] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: stopping tablet replica
I20260926 03:39:04.692386 12778 raft_consensus.cc:2229] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:39:04.693188 12778 raft_consensus.cc:2258] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:04.694253 12777 tablet_replica.cc:330] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: stopping tablet replica
I20260926 03:39:04.694738 12777 raft_consensus.cc:2229] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:39:04.695259 12779 ts_tablet_manager.cc:1918] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:04.695446 12778 ts_tablet_manager.cc:1905] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:39:04.695482 12779 log.cc:1196] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/b09160c57f474dec975e3aa2980213d2
I20260926 03:39:04.695935 12777 raft_consensus.cc:2258] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:04.695976 12779 ts_tablet_manager.cc:1939] T b09160c57f474dec975e3aa2980213d2 P cc9f3fa0a530476d80ff9d59b894ed8f: Deleting consensus metadata
I20260926 03:39:04.697849 12265 catalog_manager.cc:4841] TS cc9f3fa0a530476d80ff9d59b894ed8f (127.1.221.131:42459): tablet b09160c57f474dec975e3aa2980213d2 (table test-table [id=38d32517cbe047de85fc23436652b93b]) successfully deleted
Sep 26 03:39:04 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:39:04.699714 12777 ts_tablet_manager.cc:1905] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:39:04.703926 12778 ts_tablet_manager.cc:1918] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:04.704154 12778 log.cc:1196] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/b09160c57f474dec975e3aa2980213d2
I20260926 03:39:04.704735 12778 ts_tablet_manager.cc:1939] T b09160c57f474dec975e3aa2980213d2 P b0194615c18947e2911e9d93b72afe0f: Deleting consensus metadata
I20260926 03:39:04.706645 12777 ts_tablet_manager.cc:1918] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:04.706684 12266 catalog_manager.cc:4841] TS b0194615c18947e2911e9d93b72afe0f (127.1.221.129:34999): tablet b09160c57f474dec975e3aa2980213d2 (table test-table [id=38d32517cbe047de85fc23436652b93b]) successfully deleted
I20260926 03:39:04.706859 12777 log.cc:1196] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/b09160c57f474dec975e3aa2980213d2
I20260926 03:39:04.707237 12777 ts_tablet_manager.cc:1939] T b09160c57f474dec975e3aa2980213d2 P fe9cbdc4ebe040fba42d63089e6124e6: Deleting consensus metadata
I20260926 03:39:04.708214 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.677264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.1.221.129:34999)
0926 03:39:04.677466 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.677473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.686087 (+ 8614us) server_negotiation.cc:407] Connection header received
0926 03:39:04.686288 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.686295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.686387 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.686507 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:04.688154 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.689388 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.692530 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.693252 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.693352 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.699872 (+ 6520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:04.699890 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:04.699908 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:04.699966 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:04.703936 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.705003 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.705026 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.705030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.705246 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.705706 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.705711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.705714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.706187 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:04.706615 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.707585 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.707824 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:39:04.709007 12464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:04.709044 12264 catalog_manager.cc:4841] TS fe9cbdc4ebe040fba42d63089e6124e6 (127.1.221.130:37301): tablet b09160c57f474dec975e3aa2980213d2 (table test-table [id=38d32517cbe047de85fc23436652b93b]) successfully deleted
I20260926 03:39:04.807775 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 12341
I20260926 03:39:04.822904 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 12475
Sep 26 03:39:04 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:39:04.836861 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.823710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.1.221.130:37301)
0926 03:39:04.823894 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.823899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.823918 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:39:04.824054 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.824058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.824118 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.824198 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:04.825314 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.826008 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.827167 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.827629 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.827720 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.831692 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:04.831707 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:04.831723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:04.831760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:04.834141 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.834743 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.834750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.834752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.834888 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.835257 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.835261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.835263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.835734 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:04.836004 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.836302 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.836535 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:39:04.837522 12598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:04.924543 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 12475
I20260926 03:39:04.939617 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 12608
Sep 26 03:39:04 dist-test-slave-lqrk krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:39:04.954367 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:04.940446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.1.221.131:42459)
0926 03:39:04.940616 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:04.940620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:04.940643 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:39:04.940826 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:04.940830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:04.940903 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:04.940975 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:04.942254 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.942945 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.944190 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:04.944558 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:04.944654 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:04.949221 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:04.949238 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:04.949255 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:04.949292 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:04.952049 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.952682 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.952690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.952692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.952833 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:04.953139 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:04.953142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:04.953144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:04.953490 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:04.953718 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:04.953935 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 03:39:04.954103 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:39:04.954963 12731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:05.042357 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 12608
I20260926 03:39:05.055558 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 12248
I20260926 03:39:05.056777 12310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:05.164538 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 12248
2026-09-26T03:39:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4313 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:39:05 dist-test-slave-lqrk krb5kdc[12787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:39:05 dist-test-slave-lqrk krb5kdc[12787](info): set up 2 sockets
Sep 26 03:39:05 dist-test-slave-lqrk krb5kdc[12787](info): commencing operation
krb5kdc: starting...
W20260926 03:39:06.180102 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:39:06 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:39:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
I20260926 03:39:06.502295 1910 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmxwHQo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/postgres -l logfile start
2026-09-26 03:39:08.861 UTC [12817] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:39:08.861 UTC [12817] LOG: listening on IPv4 address "127.1.221.148", port 56465
2026-09-26 03:39:08.865 UTC [12817] LOG: listening on Unix socket "/tmp/.s.PGSQL.56465"
2026-09-26 03:39:08.884 UTC [12820] LOG: database system was shut down at 2026-09-26 03:39:07 UTC
2026-09-26 03:39:08.888 UTC [12817] LOG: database system is ready to accept connections
I20260926 03:39:09.780153 1910 mini_postgres.cc:96] Postgres bound to 56465
2026-09-26 03:39:09.796 UTC [12828] FATAL: database "slave" does not exist
127.1.221.148:56465 - accepting connections
I20260926 03:39:09.797266 1910 mini_ranger.cc:161] Starting Ranger...
I20260926 03:39:09.822031 1910 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:39:10.280650 1910 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:39:10.280768 1910 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin
2026-09-26 03:39:10,549 [I] DB FLAVOR :POSTGRES
2026-09-26 03:39:10,550 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:39:10,550 [I] Checking connection..
2026-09-26 03:39:10,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:39:10,809 [I] Checking connection passed.
2026-09-26 03:39:10,809 [I] --------- Verifying version history table ---------
2026-09-26 03:39:10,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:39:11,050 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:39:11,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:39:11,278 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:39:11,278 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:39:11,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:39:11.517 UTC [12892] WARNING: there is no transaction in progress
2026-09-26 03:39:11,523 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:39:11,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:39:11,767 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:39:11,767 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:39:11,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:39:12,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:39:12,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lqrk.c.gcp-upstream.internal','N') ;"
2026-09-26 03:39:12,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:39:12,761 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:39:12,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:39:13,020 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:39:13,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:39:13,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:39:13,498 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:39:13,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:39:14.419 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.435 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.450 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.464 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.478 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.495 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.558 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.564 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.577 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.587 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.597 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.611 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.620 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.626 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.632 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:14.639 UTC [13000] WARNING: there is no transaction in progress
2026-09-26 03:39:15,241 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:39:15,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:39:15,492 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:39:15,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:39:15,732 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:39:15,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:39:15,962 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:39:15,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:39:16,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:39:16,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lqrk.c.gcp-upstream.internal';"
2026-09-26 03:39:16,688 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:39:16,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:39:16,922 [I] DB_PATCHES have already been applied
I20260926 03:39:16.928234 1910 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmxwHQo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:39:16.928323 1910 mini_ranger.cc:193] Using host: 127.1.221.148
I20260926 03:39:16.942176 1910 mini_ranger.cc:227] Ranger admin URL: http://127.1.221.148:52799
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:39:17.950656 1910 mini_ranger.cc:160] Time spent starting Ranger: real 8.153s user 0.002s sys 0.036s
Sep 26, 2026 3:39:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:39:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:39:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:39:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:39:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:39:39.838914 1910 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.221.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:39:39 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393979, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:39:39.998375 1910 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:39:40.021401 1910 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:39:40.965477 1910 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:39:40.965641 1910 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms
I20260926 03:39:40.965770 1910 mini_ranger_kms.cc:227] Using postgres at 127.1.221.148:56465
2026-09-26 03:39:41,372 [I] DB FLAVOR :POSTGRES
2026-09-26 03:39:41,372 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:39:41,372 [I] Checking connection
2026-09-26 03:39:41,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:39:41,611 [I] connection success
2026-09-26 03:39:41,611 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:39:41,611 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:39:41,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:39:41,841 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:39:41,841 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:39:41,841 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:39:41,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:56465/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:39:42,112 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:39:42.628328 1910 mini_ranger_kms.cc:310] Created kms service
I20260926 03:39:42.879935 1910 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:39:42.929315 1910 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:39:42 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393979, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:43.332545 1910 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:39:43.332873 1910 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:39:43.332919 1910 mini_ranger_kms.cc:240] Using host: 127.1.221.148
I20260926 03:39:43.354444 1910 mini_ranger_kms.cc:276] Ranger KMS PID: 13180
I20260926 03:39:43.354614 1910 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.221.148:38637
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:39:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:39:45.485349 1910 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.487s user 0.115s sys 0.311s
I20260926 03:39:45.485596 1910 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:39:45.485780 1910 mini_ranger_kms.cc:401] 127.1.221.148:38637/kms/v1/keys
Sep 26 03:39:45 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:39:46 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Sep 26 03:39:46 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Sep 26 03:39:48 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:39:48.815733 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40501
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40501
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:38637/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:39:49.155687 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:49.155983 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:49.156095 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:49.163283 13231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:39:49.163424 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:49.163504 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:49.163578 13231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:39:49.163645 13231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:39:49.173974 13231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.221.148:38637/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40501
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:49.175436 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:49.176689 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:49.188483 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:49.188531 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:49.188544 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:49.189803 13231 server_base.cc:1034] running on GCE node
I20260926 03:39:49.190585 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:49.192178 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:49.193421 13231 hybrid_clock.cc:648] HybridClock initialized: now 1790393989193361 us; error 56 us; skew 500 ppm
I20260926 03:39:49.193782 13231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:49 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:49.202206 13231 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:39:49.204100 13231 webserver.cc:460] Webserver started at http://127.1.221.190:42417/ using document root <none> and password file <none>
I20260926 03:39:49.204931 13231 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:49.205044 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:49.205348 13231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:39:49 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:49.273957 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "d2143a6f0919426f97ef9e0bf83c7766"
format_stamp: "Formatted at 2026-09-26 03:39:49 on dist-test-slave-lqrk"
server_key: "4c2fcf622b6d12535078e3a6d2701910"
server_key_iv: "c89fb40c2fea059282ba73e3f6ae97ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:49.274947 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "d2143a6f0919426f97ef9e0bf83c7766"
format_stamp: "Formatted at 2026-09-26 03:39:49 on dist-test-slave-lqrk"
server_key: "4c2fcf622b6d12535078e3a6d2701910"
server_key_iv: "c89fb40c2fea059282ba73e3f6ae97ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:49.281395 13231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:39:49.303273 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:49.305302 13231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:39:49.305521 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "d2143a6f0919426f97ef9e0bf83c7766"
format_stamp: "Formatted at 2026-09-26 03:39:49 on dist-test-slave-lqrk"
server_key: "4c2fcf622b6d12535078e3a6d2701910"
server_key_iv: "c89fb40c2fea059282ba73e3f6ae97ed"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:49.305778 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:49.333503 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:49.339694 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:49.340080 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:49.363869 13231 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40501
I20260926 03:39:49.363893 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40501 every 8 connection(s)
I20260926 03:39:49.366632 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:39:49.371570 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:49.374297 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 13231
I20260926 03:39:49.374523 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:39:49.375459 1910 external_mini_cluster.cc:1428] Setting key 6605e548014738797a52c98cf85a333a
I20260926 03:39:49.388049 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: Bootstrap starting.
I20260926 03:39:49.392900 13298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:49.394207 13298 log.cc:824] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:49.398197 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: No bootstrap required, opened a new log
Sep 26 03:39:49 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:49.407034 13298 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } }
I20260926 03:39:49.407567 13298 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:49.407680 13298 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2143a6f0919426f97ef9e0bf83c7766, State: Initialized, Role: FOLLOWER
I20260926 03:39:49.409096 13298 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } }
I20260926 03:39:49.409430 13298 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:39:49.409612 13298 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:39:49.409821 13298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:49.413265 13298 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } }
I20260926 03:39:49.413931 13298 leader_election.cc:302] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2143a6f0919426f97ef9e0bf83c7766; no voters:
I20260926 03:39:49.414729 13298 leader_election.cc:288] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:39:49.414950 13303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:39:49.417696 13298 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:39:49.419298 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:49.378230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.1.221.190:40501)
0926 03:39:49.389011 (+ 10781us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:49.389034 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:49.389107 (+ 73us) server_negotiation.cc:407] Connection header received
0926 03:39:49.392483 (+ 3376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:49.392530 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:49.393321 (+ 791us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:49.393940 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:49.395632 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:49.397621 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:49.399735 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:49.400846 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:49.401060 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:49.406925 (+ 5865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:49.406940 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:49.406973 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:49.407050 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:49.412531 (+ 5481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:49.413589 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:49.413599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:49.413609 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:49.413815 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:49.414268 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:49.414274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:49.414277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:49.415117 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:49.416631 (+ 1514us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:49.417020 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 03:39:49.417572 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10441,"thread_start_us":10119,"threads_started":1}
I20260926 03:39:49.423288 13303 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [term 1 LEADER]: Becoming Leader. State: Replica: d2143a6f0919426f97ef9e0bf83c7766, State: Running, Role: LEADER
I20260926 03:39:49.424183 13303 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } }
I20260926 03:39:49.426627 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2143a6f0919426f97ef9e0bf83c7766. Latest consensus state: current_term: 1 leader_uuid: "d2143a6f0919426f97ef9e0bf83c7766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } } }
I20260926 03:39:49.427008 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [sys.catalog]: This master's current role is: LEADER
I20260926 03:39:49.427489 13304 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2143a6f0919426f97ef9e0bf83c7766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2143a6f0919426f97ef9e0bf83c7766" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40501 } } }
I20260926 03:39:49.427774 13304 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766 [sys.catalog]: This master's current role is: LEADER
I20260926 03:39:49.429191 13309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:39:49.443801 13309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:39:49.466300 13309 catalog_manager.cc:1347] Generated new cluster ID: 5e1b21ecd08445f688f0c5904a68adae
I20260926 03:39:49.466523 13309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:39:49.490792 13309 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:39:49.491374 13298 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:39:49.492813 13309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:39:49.501381 13309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: Generated new TSK 0
I20260926 03:39:49.508253 13309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:39:50.318786 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:38637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:39:50 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:39:50.762550 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:50.762971 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:50.763177 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:50.775218 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:50.775408 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:50.775588 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:39:50.791925 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.221.148:38637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:50.793846 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:50.795570 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:50.813946 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:50.815840 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:50.821441 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:50.821966 13343 server_base.cc:1034] running on GCE node
I20260926 03:39:50.822989 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:50.824342 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:50.825603 13343 hybrid_clock.cc:648] HybridClock initialized: now 1790393990825523 us; error 61 us; skew 500 ppm
I20260926 03:39:50.825933 13343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:50 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:50.829695 13343 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:39:50.831276 13343 webserver.cc:460] Webserver started at http://127.1.221.129:40813/ using document root <none> and password file <none>
I20260926 03:39:50.832013 13343 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:50.832127 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:50.832429 13343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:39:50 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:50.849496 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5cc12c6931f240049080010ff099cf2e"
format_stamp: "Formatted at 2026-09-26 03:39:50 on dist-test-slave-lqrk"
server_key: "578377a21b4bc2db0be79a75bbe712b3"
server_key_iv: "d14283f7936e1e2e00b5093f8f75a867"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:50.850479 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5cc12c6931f240049080010ff099cf2e"
format_stamp: "Formatted at 2026-09-26 03:39:50 on dist-test-slave-lqrk"
server_key: "578377a21b4bc2db0be79a75bbe712b3"
server_key_iv: "d14283f7936e1e2e00b5093f8f75a867"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:50.856024 13343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:39:50.876534 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:50.878315 13343 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260926 03:39:50.878578 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "5cc12c6931f240049080010ff099cf2e"
format_stamp: "Formatted at 2026-09-26 03:39:50 on dist-test-slave-lqrk"
server_key: "578377a21b4bc2db0be79a75bbe712b3"
server_key_iv: "d14283f7936e1e2e00b5093f8f75a867"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:50.878948 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:50.922933 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:50.928301 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:50.928701 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:50.929749 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:50.931268 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:50.931356 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:50.931499 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:50.931551 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:51.059783 13343 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:44129
I20260926 03:39:51.059859 13475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:44129 every 8 connection(s)
I20260926 03:39:51.061229 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:39:51.070820 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 13343
I20260926 03:39:51.071056 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:39:51.071893 1910 external_mini_cluster.cc:1428] Setting key 7da95d883161e8f121cdb05f91cd3899
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:51.138237 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:51.102524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44871 (local address 127.1.221.190:40501)
0926 03:39:51.111899 (+ 9375us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:51.111905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:51.115486 (+ 3581us) server_negotiation.cc:407] Connection header received
0926 03:39:51.115567 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:51.115572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:51.115668 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:51.115785 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:51.118227 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.119523 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.122101 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.122709 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.122819 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:51.130436 (+ 7617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:51.130445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:51.130450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:51.130494 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:51.134069 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:51.135125 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:51.135131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:51.135135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:51.135306 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:51.135906 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:51.135911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:51.135915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:51.136299 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:51.136421 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:51.137587 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0926 03:39:51.137849 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9186,"thread_start_us":9000,"threads_started":1}
I20260926 03:39:51.140803 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:51.103197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40501 (local address 127.1.221.129:44871)
0926 03:39:51.112322 (+ 9125us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:51.112386 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:51.114945 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:51.116300 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:51.116315 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:51.117020 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:51.118010 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:51.118038 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.119750 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.119756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:51.121895 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:51.121916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.123173 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.123179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:51.123502 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:51.124283 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:51.124315 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:51.130116 (+ 5801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:51.134290 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:51.134300 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:51.134322 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:51.134978 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:51.135486 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:51.135491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:51.135494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:51.135771 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:51.136542 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:51.136551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:51.137062 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 03:39:51.138787 (+ 1725us) client_negotiation.cc:240] Negotiation successful
0926 03:39:51.139358 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8761,"thread_start_us":8475,"threads_started":1}
I20260926 03:39:51.142967 13476 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40501
I20260926 03:39:51.143299 13476 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:51.144415 13476 heartbeater.cc:511] Master 127.1.221.190:40501 requested a full tablet report, sending...
I20260926 03:39:51.147109 13263 ts_manager.cc:194] Registered new tserver with Master: 5cc12c6931f240049080010ff099cf2e (127.1.221.129:44129)
I20260926 03:39:51.149094 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:44871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:39:51.162540 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:38637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
W20260926 03:39:51.630196 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:51.630517 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:51.630625 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:51.638402 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:51.638514 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:51.638664 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:39:51.648555 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.221.148:38637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:51.649847 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:51.651130 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:51.664786 13499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:51.664776 13497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:51.665073 13496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:51.665587 13482 server_base.cc:1034] running on GCE node
I20260926 03:39:51.666329 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:51.667408 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:51.668649 13482 hybrid_clock.cc:648] HybridClock initialized: now 1790393991668612 us; error 33 us; skew 500 ppm
I20260926 03:39:51.669014 13482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393991, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:51.672324 13482 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:39:51.673811 13482 webserver.cc:460] Webserver started at http://127.1.221.130:44499/ using document root <none> and password file <none>
I20260926 03:39:51.674484 13482 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:51.674588 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:51.674912 13482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393991, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:51.691497 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5"
format_stamp: "Formatted at 2026-09-26 03:39:51 on dist-test-slave-lqrk"
server_key: "37d77db4e302fb27690c95b79959ee46"
server_key_iv: "9a3580475e34fd212a7b44d6596aa9cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:51.692443 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5"
format_stamp: "Formatted at 2026-09-26 03:39:51 on dist-test-slave-lqrk"
server_key: "37d77db4e302fb27690c95b79959ee46"
server_key_iv: "9a3580475e34fd212a7b44d6596aa9cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:51.697962 13482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:39:51.712986 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:51.714920 13482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 03:39:51.715116 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5"
format_stamp: "Formatted at 2026-09-26 03:39:51 on dist-test-slave-lqrk"
server_key: "37d77db4e302fb27690c95b79959ee46"
server_key_iv: "9a3580475e34fd212a7b44d6596aa9cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:51.715415 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:51.737880 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:51.743177 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:51.743574 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:51.744831 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:51.746678 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:51.746776 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:51.746912 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:51.746973 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:51.781486 13482 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:40497
I20260926 03:39:51.781504 13618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:40497 every 8 connection(s)
I20260926 03:39:51.783190 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:39:51.783881 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 13482
I20260926 03:39:51.784045 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:39:51.784582 1910 external_mini_cluster.cc:1428] Setting key 1dfd579ec928d10d4326bf9db373c46c
Sep 26 03:39:51 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393991, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:39:51.818887 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:51.787741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56471 (local address 127.1.221.190:40501)
0926 03:39:51.795832 (+ 8091us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:51.795840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:51.801753 (+ 5913us) server_negotiation.cc:407] Connection header received
0926 03:39:51.801855 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:51.801861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:51.801944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:51.802040 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:51.803997 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.805146 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.807491 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.808128 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.808263 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:51.812677 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:51.812682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:51.812685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:51.812718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:51.815775 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:51.816533 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:51.816537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:51.816539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:51.816664 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:51.817112 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:51.817115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:51.817118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:51.817394 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:51.817520 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:51.818286 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 03:39:51.818514 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7937,"thread_start_us":7803,"threads_started":1}
I20260926 03:39:51.820005 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:51.787101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40501 (local address 127.1.221.130:56471)
0926 03:39:51.796360 (+ 9259us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:51.796421 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:51.800728 (+ 4307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:51.802320 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:51.802332 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:51.802910 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:51.803773 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:51.803800 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.805312 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.805318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:51.807276 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:51.807302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:51.808417 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:51.808426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:51.808753 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:51.809602 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:51.809634 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:51.812394 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:51.815930 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:51.815940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:51.815960 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:51.816409 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:51.816764 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:51.816769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:51.816772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:51.816997 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:51.817523 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:51.817531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:51.817867 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 03:39:51.818102 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:39:51.818694 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8896,"thread_start_us":12227,"threads_started":1}
I20260926 03:39:51.821462 13619 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40501
I20260926 03:39:51.821770 13619 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:51.822511 13619 heartbeater.cc:511] Master 127.1.221.190:40501 requested a full tablet report, sending...
I20260926 03:39:51.823894 13263 ts_manager.cc:194] Registered new tserver with Master: 2c24ca0ebcee440c911fb8cb7fa1cbe5 (127.1.221.130:40497)
I20260926 03:39:51.824929 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:56471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:39:51.844180 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:38637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:39:52.152990 13476 heartbeater.cc:503] Master 127.1.221.190:40501 was elected leader, sending a full tablet report...
W20260926 03:39:52.186480 13625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:39:52.186766 13625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:39:52.186882 13625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:39:52.194039 13625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:39:52.194149 13625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:39:52.194293 13625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:39:52.205029 13625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.221.148:38637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:39:52.206311 13625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:39:52.207612 13625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:39:52.220835 13633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:52.220934 13631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:39:52.220835 13630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:39:52.222045 13625 server_base.cc:1034] running on GCE node
I20260926 03:39:52.222632 13625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:39:52.223568 13625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:39:52.224777 13625 hybrid_clock.cc:648] HybridClock initialized: now 1790393992224731 us; error 37 us; skew 500 ppm
I20260926 03:39:52.225088 13625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:39:52 dist-test-slave-lqrk krb5kdc[12787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393992, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:39:52.228113 13625 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:39:52.229543 13625 webserver.cc:460] Webserver started at http://127.1.221.131:35861/ using document root <none> and password file <none>
I20260926 03:39:52.230171 13625 fs_manager.cc:359] Metadata directory not provided
I20260926 03:39:52.230273 13625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:39:52.230522 13625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:39:52 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393992, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:39:52.247474 13625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad"
format_stamp: "Formatted at 2026-09-26 03:39:52 on dist-test-slave-lqrk"
server_key: "3d073d127bc1fd59457e1a060838ad70"
server_key_iv: "b3f7f302c20d7ccc212bac4dfd8cf40d"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:52.248693 13625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad"
format_stamp: "Formatted at 2026-09-26 03:39:52 on dist-test-slave-lqrk"
server_key: "3d073d127bc1fd59457e1a060838ad70"
server_key_iv: "b3f7f302c20d7ccc212bac4dfd8cf40d"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:52.256119 13625 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 03:39:52.273953 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:52.275789 13625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:39:52.275985 13625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad"
format_stamp: "Formatted at 2026-09-26 03:39:52 on dist-test-slave-lqrk"
server_key: "3d073d127bc1fd59457e1a060838ad70"
server_key_iv: "b3f7f302c20d7ccc212bac4dfd8cf40d"
server_key_version: "kuduclusterkey@0"
I20260926 03:39:52.276202 13625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:39:52.287667 13625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:39:52.291755 13625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:39:52.292045 13625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:39:52.293008 13625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:39:52.294363 13625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:39:52.294432 13625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:52.294540 13625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:39:52.294589 13625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:39:52.327339 13625 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:37819
I20260926 03:39:52.327368 13752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:37819 every 8 connection(s)
I20260926 03:39:52.328647 13625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:39:52.329905 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 13625
I20260926 03:39:52.330068 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:39:52.330523 1910 external_mini_cluster.cc:1428] Setting key 172d173851ebd7736f54302c2212875a
Sep 26 03:39:52 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393992, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:39:52.349807 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.331512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37553 (local address 127.1.221.190:40501)
0926 03:39:52.332008 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.332012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.333407 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 03:39:52.335288 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.335293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.335425 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.335527 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:52.337103 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.337840 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.339462 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.339904 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.340015 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.343364 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:52.343369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:52.343373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:52.343444 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:52.346492 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:52.347481 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:52.347487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:52.347489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:52.347619 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:52.348032 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:52.348036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:52.348038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:52.348338 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:52.348447 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.349301 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.349496 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":262,"threads_started":1}
I20260926 03:39:52.350783 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.332019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40501 (local address 127.1.221.131:37553)
0926 03:39:52.333244 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.333291 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.335013 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.335827 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.335837 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.336307 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:39:52.336927 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.336945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.337984 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.337988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.339276 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.339296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.340123 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.340131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.340387 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.341039 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:39:52.341064 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:39:52.343117 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:39:52.346864 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:52.346871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:52.346888 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:52.347353 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:52.347720 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:39:52.347724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:39:52.347726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:39:52.347934 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:39:52.348498 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:39:52.348507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:39:52.348878 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.349123 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.349527 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":756,"threads_started":1}
I20260926 03:39:52.352576 13753 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40501
I20260926 03:39:52.352986 13753 heartbeater.cc:464] Registering TS with master...
I20260926 03:39:52.353782 13753 heartbeater.cc:511] Master 127.1.221.190:40501 requested a full tablet report, sending...
I20260926 03:39:52.355288 13263 ts_manager.cc:194] Registered new tserver with Master: ec9e91b8909c4e41ab67ea1b9b1f35ad (127.1.221.131:37819)
I20260926 03:39:52.356174 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:37553
I20260926 03:39:52.359140 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:39:52.377291 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.363925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.1.221.190:40501)
0926 03:39:52.364104 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.364111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.365064 (+ 953us) server_negotiation.cc:407] Connection header received
0926 03:39:52.365272 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.365277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.365369 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.365455 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:52.366846 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.367674 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.369020 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.369456 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.369564 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.372024 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:52.372029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:52.372033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:52.372067 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:52.374577 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:52.375291 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:52.375297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:52.375300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:52.375475 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:52.375842 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:52.375848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:52.375852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:52.376151 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:52.376391 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.376714 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.376960 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:39:52.381824 13263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:39:52.389912 13263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:39:52.418936 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.403808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.1.221.129:44129)
0926 03:39:52.407587 (+ 3779us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.407598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.408460 (+ 862us) server_negotiation.cc:407] Connection header received
0926 03:39:52.408745 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.408752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.409021 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.409184 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:52.410416 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.412137 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.416431 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.417756 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.417881 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.418119 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.418391 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.418547 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2211,"thread_start_us":2038,"threads_started":1}
I20260926 03:39:52.419008 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.402974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:37819 (local address 127.0.0.1:51778)
0926 03:39:52.407320 (+ 4346us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.407360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.407572 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.408704 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.408711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.408738 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:52.409156 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.409169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.412704 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.412710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.416253 (+ 3543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.416266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.418109 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.418125 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.418314 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.418347 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.418410 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.418541 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4088,"mutex_wait_us":95,"thread_start_us":5128,"threads_started":1}
I20260926 03:39:52.419032 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.403109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:44129 (local address 127.0.0.1:33352)
0926 03:39:52.408349 (+ 5240us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.408366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.408574 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.409474 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.409478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.409499 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:52.410245 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.410258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.412312 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.412317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.416247 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.416263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.417954 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.417960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.418157 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.418219 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.418327 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.418534 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5134,"mutex_wait_us":30,"thread_start_us":1896,"threads_started":1}
I20260926 03:39:52.420332 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.403266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.1.221.131:37819)
0926 03:39:52.408034 (+ 4768us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.408042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.408075 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:39:52.408172 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.408180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.408454 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.409368 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:52.409615 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.412553 (+ 2938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.416433 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.417885 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.419240 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.419745 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.419812 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.419964 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4612,"thread_start_us":4463,"threads_started":1}
I20260926 03:39:52.424005 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.405493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.1.221.130:40497)
0926 03:39:52.416369 (+ 10876us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.416378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.416406 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:39:52.416503 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.416511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.416797 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.417026 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:52.417757 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.419196 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.421930 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.422976 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.423101 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.423301 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.423533 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.423688 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10714,"thread_start_us":10602,"threads_started":1}
I20260926 03:39:52.423508 13411 tablet_service.cc:1467] Processing CreateTablet for tablet faaee9bbb6744436b36381e32629cedf (DEFAULT_TABLE table=test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:52.423597 13688 tablet_service.cc:1467] Processing CreateTablet for tablet faaee9bbb6744436b36381e32629cedf (DEFAULT_TABLE table=test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:52.425055 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaee9bbb6744436b36381e32629cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:52.425055 13411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaee9bbb6744436b36381e32629cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:52.428005 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.405055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40497 (local address 127.0.0.1:43264)
0926 03:39:52.415751 (+ 10696us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.415769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.415881 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.417018 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.417023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.417051 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:52.417403 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.417414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.419351 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.419356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.421788 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.421801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.423186 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.423192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.423351 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.423403 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.423490 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.427467 (+ 3977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10528,"thread_start_us":10386,"threads_started":1}
I20260926 03:39:52.432301 13554 tablet_service.cc:1467] Processing CreateTablet for tablet faaee9bbb6744436b36381e32629cedf (DEFAULT_TABLE table=test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:39:52.433801 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faaee9bbb6744436b36381e32629cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:39:52.438773 13772 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Bootstrap starting.
I20260926 03:39:52.442704 13772 tablet_bootstrap.cc:654] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:52.444099 13772 log.cc:824] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:52.446806 13775 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Bootstrap starting.
I20260926 03:39:52.447088 13773 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Bootstrap starting.
I20260926 03:39:52.447405 13772 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: No bootstrap required, opened a new log
I20260926 03:39:52.448169 13772 ts_tablet_manager.cc:1397] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 03:39:52.450935 13775 tablet_bootstrap.cc:654] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:52.451273 13773 tablet_bootstrap.cc:654] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Neither blocks nor log segments found. Creating new log.
I20260926 03:39:52.452554 13773 log.cc:824] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:52.452715 13775 log.cc:824] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:39:52.455258 13773 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: No bootstrap required, opened a new log
I20260926 03:39:52.455519 13773 ts_tablet_manager.cc:1397] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:39:52.455865 13772 raft_consensus.cc:348] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.456280 13772 raft_consensus.cc:374] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:52.456377 13772 raft_consensus.cc:729] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc12c6931f240049080010ff099cf2e, State: Initialized, Role: FOLLOWER
I20260926 03:39:52.456967 13772 consensus_queue.cc:260] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.458438 13772 ts_tablet_manager.cc:1428] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 03:39:52.457810 13775 tablet_bootstrap.cc:492] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: No bootstrap required, opened a new log
I20260926 03:39:52.459474 13775 ts_tablet_manager.cc:1397] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 03:39:52.465554 13773 raft_consensus.cc:348] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.465997 13773 raft_consensus.cc:374] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:52.466095 13773 raft_consensus.cc:729] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9e91b8909c4e41ab67ea1b9b1f35ad, State: Initialized, Role: FOLLOWER
I20260926 03:39:52.466712 13773 consensus_queue.cc:260] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.467810 13753 heartbeater.cc:503] Master 127.1.221.190:40501 was elected leader, sending a full tablet report...
I20260926 03:39:52.468519 13773 ts_tablet_manager.cc:1428] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 03:39:52.469522 13775 raft_consensus.cc:348] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.469897 13775 raft_consensus.cc:374] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:39:52.469995 13775 raft_consensus.cc:729] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c24ca0ebcee440c911fb8cb7fa1cbe5, State: Initialized, Role: FOLLOWER
I20260926 03:39:52.470551 13775 consensus_queue.cc:260] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.471596 13619 heartbeater.cc:503] Master 127.1.221.190:40501 was elected leader, sending a full tablet report...
I20260926 03:39:52.472128 13775 ts_tablet_manager.cc:1428] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260926 03:39:52.552549 13620 tablet.cc:2369] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:39:52.580237 13754 tablet.cc:2369] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:39:52.593626 13477 tablet.cc:2369] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:39:52.901322 13779 raft_consensus.cc:482] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:39:52.901640 13779 raft_consensus.cc:504] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.903776 13779 leader_election.cc:288] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc12c6931f240049080010ff099cf2e (127.1.221.129:44129), 2c24ca0ebcee440c911fb8cb7fa1cbe5 (127.1.221.130:40497)
I20260926 03:39:52.911468 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.904313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50837 (local address 127.1.221.129:44129)
0926 03:39:52.904530 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.904537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.904886 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:39:52.905224 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.905231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.905309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.905372 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:52.906432 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.907574 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.909500 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.910517 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.910686 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.910765 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.911052 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.911183 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:39:52.911909 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.904005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:44129 (local address 127.1.221.131:50837)
0926 03:39:52.904756 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.904777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.905011 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.905597 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.905602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.905701 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:52.906185 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.906200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.907682 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.907688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.909330 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.909344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.910685 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.910692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.910883 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.910921 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.911019 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.911166 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":28,"thread_start_us":737,"threads_started":1}
I20260926 03:39:52.912501 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.904326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:52841 (local address 127.1.221.130:40497)
0926 03:39:52.904744 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.904751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.905370 (+ 619us) server_negotiation.cc:407] Connection header received
0926 03:39:52.905540 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.905545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.905624 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.905704 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:39:52.906336 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.907484 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.910246 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.911448 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.911592 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.911659 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.912119 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.912260 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260926 03:39:52.913103 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.904005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40497 (local address 127.1.221.131:52841)
0926 03:39:52.905268 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0926 03:39:52.905283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:39:52.905411 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:39:52.905742 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:39:52.905746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:39:52.905782 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:39:52.906190 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.906199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.907671 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.907677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.909997 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:39:52.910039 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.911788 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.911793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:39:52.911931 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.911955 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:39:52.912721 (+ 766us) client_negotiation.cc:240] Negotiation successful
0926 03:39:52.912846 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":379,"thread_start_us":646,"threads_started":1}
I20260926 03:39:52.913132 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaee9bbb6744436b36381e32629cedf" candidate_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc12c6931f240049080010ff099cf2e" is_pre_election: true
I20260926 03:39:52.913831 13431 raft_consensus.cc:2454] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec9e91b8909c4e41ab67ea1b9b1f35ad in term 0.
I20260926 03:39:52.914574 13642 leader_election.cc:302] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc12c6931f240049080010ff099cf2e, ec9e91b8909c4e41ab67ea1b9b1f35ad; no voters:
I20260926 03:39:52.914938 13779 raft_consensus.cc:2781] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:39:52.914825 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaee9bbb6744436b36381e32629cedf" candidate_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" is_pre_election: true
I20260926 03:39:52.915068 13779 raft_consensus.cc:482] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:39:52.915158 13779 raft_consensus.cc:3037] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:52.915426 13574 raft_consensus.cc:2454] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec9e91b8909c4e41ab67ea1b9b1f35ad in term 0.
I20260926 03:39:52.917522 13779 raft_consensus.cc:504] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.918145 13779 leader_election.cc:288] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [CANDIDATE]: Term 1 election: Requested vote from peers 5cc12c6931f240049080010ff099cf2e (127.1.221.129:44129), 2c24ca0ebcee440c911fb8cb7fa1cbe5 (127.1.221.130:40497)
I20260926 03:39:52.918622 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaee9bbb6744436b36381e32629cedf" candidate_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc12c6931f240049080010ff099cf2e"
I20260926 03:39:52.918676 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faaee9bbb6744436b36381e32629cedf" candidate_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5"
I20260926 03:39:52.918857 13431 raft_consensus.cc:3037] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:52.918849 13574 raft_consensus.cc:3037] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:39:52.921993 13574 raft_consensus.cc:2454] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec9e91b8909c4e41ab67ea1b9b1f35ad in term 1.
I20260926 03:39:52.922084 13431 raft_consensus.cc:2454] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec9e91b8909c4e41ab67ea1b9b1f35ad in term 1.
I20260926 03:39:52.922513 13641 leader_election.cc:302] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c24ca0ebcee440c911fb8cb7fa1cbe5, ec9e91b8909c4e41ab67ea1b9b1f35ad; no voters:
I20260926 03:39:52.922847 13779 raft_consensus.cc:2781] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:39:52.923264 13779 raft_consensus.cc:686] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 1 LEADER]: Becoming Leader. State: Replica: ec9e91b8909c4e41ab67ea1b9b1f35ad, State: Running, Role: LEADER
I20260926 03:39:52.923779 13779 consensus_queue.cc:237] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } }
I20260926 03:39:52.928535 13263 catalog_manager.cc:5495] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ec9e91b8909c4e41ab67ea1b9b1f35ad (127.1.221.131). New cstate: current_term: 1 leader_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec9e91b8909c4e41ab67ea1b9b1f35ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 } health_report { overall_health: UNKNOWN } } }
I20260926 03:39:52.979668 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:52.973869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.1.221.131:37819)
0926 03:39:52.974323 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:52.974328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:52.974475 (+ 147us) server_negotiation.cc:407] Connection header received
0926 03:39:52.974649 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:52.974653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:52.974738 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:52.974873 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:39:52.975251 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.976012 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.977458 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:52.977860 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:52.977958 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:52.978479 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:39:52.979037 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:39:52.979143 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:52.979312 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 03:39:52.979404 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":227,"threads_started":1}
I20260926 03:39:52.987490 13574 raft_consensus.cc:1261] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 1 FOLLOWER]: Refusing update from remote peer ec9e91b8909c4e41ab67ea1b9b1f35ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:39:52.987491 13431 raft_consensus.cc:1261] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 1 FOLLOWER]: Refusing update from remote peer ec9e91b8909c4e41ab67ea1b9b1f35ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:39:52.988379 13783 consensus_queue.cc:1035] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc12c6931f240049080010ff099cf2e" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:39:52.988693 13779 consensus_queue.cc:1035] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c24ca0ebcee440c911fb8cb7fa1cbe5" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:39:53.004822 13795 mvcc.cc:205] Tried to move back new op lower bound from 7333453795267723264 to 7333453795019448320. Current Snapshot: MvccSnapshot[applied={T|T < 7333453795267723264}]
I20260926 03:39:53.013163 13262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:39:53.013453 13262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:39:53.017055 13262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2143a6f0919426f97ef9e0bf83c7766: Sending DeleteTablet for 3 replicas of tablet faaee9bbb6744436b36381e32629cedf
I20260926 03:39:53.018185 13688 tablet_service.cc:1514] Processing DeleteTablet for tablet faaee9bbb6744436b36381e32629cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:53 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:51778
I20260926 03:39:53.018219 13411 tablet_service.cc:1514] Processing DeleteTablet for tablet faaee9bbb6744436b36381e32629cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:53 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33352
I20260926 03:39:53.018481 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet faaee9bbb6744436b36381e32629cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:39:53 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43264
I20260926 03:39:53.020570 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 13343
I20260926 03:39:53.027109 13803 tablet_replica.cc:330] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: stopping tablet replica
I20260926 03:39:53.027592 13803 raft_consensus.cc:2229] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:39:53.028182 13803 raft_consensus.cc:2258] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:53.030349 13803 ts_tablet_manager.cc:1905] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:39:53.032024 13804 tablet_replica.cc:330] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: stopping tablet replica
I20260926 03:39:53.032591 13804 raft_consensus.cc:2229] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:39:53.033568 13804 raft_consensus.cc:2258] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:53.036226 13804 ts_tablet_manager.cc:1905] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:39:53.037451 13803 ts_tablet_manager.cc:1918] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:53.037613 13803 log.cc:1196] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/faaee9bbb6744436b36381e32629cedf
I20260926 03:39:53.038115 13803 ts_tablet_manager.cc:1939] T faaee9bbb6744436b36381e32629cedf P 5cc12c6931f240049080010ff099cf2e: Deleting consensus metadata
I20260926 03:39:53.040053 13248 catalog_manager.cc:4841] TS 5cc12c6931f240049080010ff099cf2e (127.1.221.129:44129): tablet faaee9bbb6744436b36381e32629cedf (table test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]) successfully deleted
I20260926 03:39:53.040630 13802 tablet_replica.cc:330] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: stopping tablet replica
I20260926 03:39:53.041010 13802 raft_consensus.cc:2229] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:39:53.041560 13802 raft_consensus.cc:2258] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:39:53.043506 13802 ts_tablet_manager.cc:1905] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:39:53 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:39:53.047230 13804 ts_tablet_manager.cc:1918] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:53.047410 13804 log.cc:1196] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/faaee9bbb6744436b36381e32629cedf
I20260926 03:39:53.047878 13804 ts_tablet_manager.cc:1939] T faaee9bbb6744436b36381e32629cedf P 2c24ca0ebcee440c911fb8cb7fa1cbe5: Deleting consensus metadata
I20260926 03:39:53.049734 13247 catalog_manager.cc:4841] TS 2c24ca0ebcee440c911fb8cb7fa1cbe5 (127.1.221.130:40497): tablet faaee9bbb6744436b36381e32629cedf (table test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]) successfully deleted
I20260926 03:39:53.051226 13802 ts_tablet_manager.cc:1918] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:39:53.051421 13802 log.cc:1196] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/faaee9bbb6744436b36381e32629cedf
I20260926 03:39:53.051910 13802 ts_tablet_manager.cc:1939] T faaee9bbb6744436b36381e32629cedf P ec9e91b8909c4e41ab67ea1b9b1f35ad: Deleting consensus metadata
I20260926 03:39:53.053720 13249 catalog_manager.cc:4841] TS ec9e91b8909c4e41ab67ea1b9b1f35ad (127.1.221.131:37819): tablet faaee9bbb6744436b36381e32629cedf (table test-table [id=c2a23e011b4f4fa19abd4a4e2d1bd241]) successfully deleted
I20260926 03:39:53.055915 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:53.022004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.1.221.129:44129)
0926 03:39:53.022183 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:53.022189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:53.036260 (+ 14071us) server_negotiation.cc:407] Connection header received
0926 03:39:53.036542 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:53.036548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:53.036638 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:53.036710 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:53.038546 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.039700 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.041742 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.042860 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.042967 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:53.048723 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:53.048742 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:53.048761 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:53.048803 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:53.053125 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.054007 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.054015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.054017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.054135 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.054515 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.054518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.054520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.054876 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:53.055090 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:53.055485 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:39:53.055646 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:39:53.056583 13471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:53.156544 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 13343
I20260926 03:39:53.170966 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 13482
Sep 26 03:39:53 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:39:53.188369 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:53.171771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.1.221.130:40497)
0926 03:39:53.172014 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:53.172020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:53.172044 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:39:53.172112 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:53.172116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:53.172189 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:53.172289 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:53.173510 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.174677 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.176070 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.176752 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.176905 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:53.181545 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:53.181566 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:53.181587 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:53.181633 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:53.185356 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.186078 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.186089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.186093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.186289 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.186689 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.186694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.186697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.187168 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:53.187480 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:53.187769 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 03:39:53.187986 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:39:53.189111 13614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:53.276597 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 13482
I20260926 03:39:53.290071 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 13625
Sep 26 03:39:53 dist-test-slave-lqrk krb5kdc[12787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:39:53.307616 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:39:53.290906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.1.221.131:37819)
0926 03:39:53.291083 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:39:53.291089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:39:53.291111 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:39:53.291246 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:39:53.291252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:39:53.291330 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:39:53.291514 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:39:53.292892 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.293706 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.294977 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:39:53.295446 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:39:53.295560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:39:53.300202 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:39:53.300215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:39:53.300232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:39:53.300273 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:39:53.304483 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.305290 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.305301 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.305305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.305487 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:39:53.305974 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:39:53.305978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:39:53.305980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:39:53.306354 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:39:53.306654 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:39:53.306951 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:39:53.307221 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:39:53.308410 13748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:53.434093 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 13625
I20260926 03:39:53.449190 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 13231
I20260926 03:39:53.450325 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:39:53.559839 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 13231
I20260926 03:39:53.576975 1910 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:39:53.912599 1910 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:39:53.912842 1910 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:39:54.250437 1910 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:39:54.250 UTC [12817] LOG: received smart shutdown request
2026-09-26 03:39:54.254 UTC [12817] LOG: background worker "logical replication launcher" (PID 12826) exited with exit code 1
2026-09-26 03:39:54.254 UTC [12821] LOG: shutting down
2026-09-26 03:39:54.436 UTC [12817] LOG: database system is shut down
2026-09-26T03:39:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49390 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:39:54 dist-test-slave-lqrk krb5kdc[13818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:39:54 dist-test-slave-lqrk krb5kdc[13818](info): set up 2 sockets
Sep 26 03:39:54 dist-test-slave-lqrk krb5kdc[13818](info): commencing operation
krb5kdc: starting...
W20260926 03:39:55.592067 1910 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.009s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:39:55 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:39:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
I20260926 03:39:55.958843 1910 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmxwHQo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/postgres -l logfile start
2026-09-26 03:39:58.521 UTC [13848] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:39:58.521 UTC [13848] LOG: listening on IPv4 address "127.1.221.148", port 47715
2026-09-26 03:39:58.525 UTC [13848] LOG: listening on Unix socket "/tmp/.s.PGSQL.47715"
2026-09-26 03:39:58.545 UTC [13851] LOG: database system was shut down at 2026-09-26 03:39:57 UTC
2026-09-26 03:39:58.549 UTC [13848] LOG: database system is ready to accept connections
I20260926 03:39:59.443532 1910 mini_postgres.cc:96] Postgres bound to 47715
2026-09-26 03:39:59.468 UTC [13859] FATAL: database "slave" does not exist
127.1.221.148:47715 - accepting connections
I20260926 03:39:59.469373 1910 mini_ranger.cc:161] Starting Ranger...
I20260926 03:39:59.499524 1910 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:39:59.997555 1910 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:39:59.997701 1910 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin
2026-09-26 03:40:00,292 [I] DB FLAVOR :POSTGRES
2026-09-26 03:40:00,293 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:40:00,293 [I] Checking connection..
2026-09-26 03:40:00,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:40:00,572 [I] Checking connection passed.
2026-09-26 03:40:00,573 [I] --------- Verifying version history table ---------
2026-09-26 03:40:00,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:40:00,847 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:40:00,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:40:01,129 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:40:01,129 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:40:01,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:40:01.381 UTC [13923] WARNING: there is no transaction in progress
2026-09-26 03:40:01,391 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:40:01,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:40:01,656 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:40:01,656 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:40:01,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:40:01,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:40:02,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lqrk.c.gcp-upstream.internal','N') ;"
2026-09-26 03:40:02,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:40:02,677 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:40:02,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:40:02,914 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:40:02,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:40:03,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:40:03,391 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:40:03,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:40:04.351 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.369 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.385 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.401 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.417 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.438 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.505 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.512 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.527 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.538 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.550 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.565 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.576 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.584 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.592 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:04.600 UTC [14031] WARNING: there is no transaction in progress
2026-09-26 03:40:05,257 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:40:05,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:40:05,506 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:40:05,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:40:05,742 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:40:05,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:40:06,000 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:40:06,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:40:06,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:40:06,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lqrk.c.gcp-upstream.internal';"
2026-09-26 03:40:06,731 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:40:06,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:40:07,010 [I] DB_PATCHES have already been applied
I20260926 03:40:07.016299 1910 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmxwHQo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:40:07.016417 1910 mini_ranger.cc:193] Using host: 127.1.221.148
I20260926 03:40:07.032905 1910 mini_ranger.cc:227] Ranger admin URL: http://127.1.221.148:59005
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:40:08.053680 1910 mini_ranger.cc:160] Time spent starting Ranger: real 8.584s user 0.003s sys 0.053s
Sep 26, 2026 3:40:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:40:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:40:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:40:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:40:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:40:28.920977 1910 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.221.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:40:29 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394029, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:40:29.122886 1910 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:40:29.143298 1910 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:40:30.053583 1910 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:40:30.053704 1910 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms
I20260926 03:40:30.053746 1910 mini_ranger_kms.cc:227] Using postgres at 127.1.221.148:47715
2026-09-26 03:40:30,453 [I] DB FLAVOR :POSTGRES
2026-09-26 03:40:30,453 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:40:30,453 [I] Checking connection
2026-09-26 03:40:30,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:40:30,713 [I] connection success
2026-09-26 03:40:30,714 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:40:30,714 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:40:30,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:40:30,963 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:40:30,963 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:40:30,964 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:40:30,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:47715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:40:31,227 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:40:31.730345 1910 mini_ranger_kms.cc:310] Created kms service
I20260926 03:40:31.912631 1910 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:40:31.956535 1910 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:40:31 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394029, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:32.349977 1910 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:40:32.350289 1910 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmxwHQo/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmxwHQo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:40:32.350337 1910 mini_ranger_kms.cc:240] Using host: 127.1.221.148
I20260926 03:40:32.368402 1910 mini_ranger_kms.cc:276] Ranger KMS PID: 14211
I20260926 03:40:32.368633 1910 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.221.148:47023
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:40:34.442999 1910 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.320s user 0.089s sys 0.319s
I20260926 03:40:34.443228 1910 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:40:34.443406 1910 mini_ranger_kms.cc:401] 127.1.221.148:47023/kms/v1/keys
Sep 26 03:40:34 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:40:35 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Sep 26 03:40:35 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Sep 26 03:40:38 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:40:38.105052 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43715
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43715
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:38.442751 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:38.443087 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:38.443223 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:38.450814 14262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:38.450974 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:38.451052 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:38.451129 14262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:38.451195 14262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:38.463169 14262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.221.148:47023/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43715
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:38.464547 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:38.465875 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:38.478520 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:38.480512 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:38.480988 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:38.484736 14262 server_base.cc:1034] running on GCE node
I20260926 03:40:38.485589 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:38.487056 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:38.488296 14262 hybrid_clock.cc:648] HybridClock initialized: now 1790394038488241 us; error 32 us; skew 500 ppm
I20260926 03:40:38.488709 14262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:38 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:38.497359 14262 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:40:38.499042 14262 webserver.cc:460] Webserver started at http://127.1.221.190:45121/ using document root <none> and password file <none>
I20260926 03:40:38.499809 14262 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:38.499908 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:38.500170 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:40:38 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:38.561609 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "c03fe24467d94db6bfeaaf9e1c88ab35"
format_stamp: "Formatted at 2026-09-26 03:40:38 on dist-test-slave-lqrk"
server_key: "973ef730e5bb86cbe43e5cd166d9a386"
server_key_iv: "8947ab65629bd5bd16ba4a92bf293a0d"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:38.562825 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "c03fe24467d94db6bfeaaf9e1c88ab35"
format_stamp: "Formatted at 2026-09-26 03:40:38 on dist-test-slave-lqrk"
server_key: "973ef730e5bb86cbe43e5cd166d9a386"
server_key_iv: "8947ab65629bd5bd16ba4a92bf293a0d"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:38.570289 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 03:40:38.598320 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:38.601647 14262 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 03:40:38.601951 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "c03fe24467d94db6bfeaaf9e1c88ab35"
format_stamp: "Formatted at 2026-09-26 03:40:38 on dist-test-slave-lqrk"
server_key: "973ef730e5bb86cbe43e5cd166d9a386"
server_key_iv: "8947ab65629bd5bd16ba4a92bf293a0d"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:38.602253 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:38.631724 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:38.635742 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:38.636053 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:38.653812 14262 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43715
I20260926 03:40:38.653818 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43715 every 8 connection(s)
I20260926 03:40:38.655166 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:40:38.658977 14329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:38.664069 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14262
I20260926 03:40:38.664299 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:40:38.665095 1910 external_mini_cluster.cc:1428] Setting key bd14dd1acf91ace1ce1476fb4cf389ac
I20260926 03:40:38.670531 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Bootstrap starting.
I20260926 03:40:38.674279 14329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:38.675552 14329 log.cc:824] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:38.679039 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: No bootstrap required, opened a new log
Sep 26 03:40:38 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:38.688000 14329 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:38.688380 14329 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:38.688475 14329 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03fe24467d94db6bfeaaf9e1c88ab35, State: Initialized, Role: FOLLOWER
I20260926 03:40:38.688504 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:38.667290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.1.221.190:43715)
0926 03:40:38.668141 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:38.668152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:38.668197 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:40:38.669917 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:38.669941 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:38.670346 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:38.670724 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:38.672351 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:38.673678 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:38.675492 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:38.676343 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:38.676548 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:38.681037 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:38.681045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:38.681064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:38.681111 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:38.684258 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:38.685122 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:38.685129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:38.685136 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:38.685291 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:38.685668 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:38.685671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:38.685676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:38.686250 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:38.686489 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:38.686908 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:40:38.687279 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":389,"threads_started":1}
I20260926 03:40:38.689100 14329 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:38.689332 14329 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:40:38.689459 14329 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:40:38.689612 14329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:38.691866 14329 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:38.692335 14329 leader_election.cc:302] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03fe24467d94db6bfeaaf9e1c88ab35; no voters:
I20260926 03:40:38.692880 14329 leader_election.cc:288] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:40:38.693013 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:38.693622 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 LEADER]: Becoming Leader. State: Replica: c03fe24467d94db6bfeaaf9e1c88ab35, State: Running, Role: LEADER
I20260926 03:40:38.694180 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:38.694849 14329 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:38.696851 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03fe24467d94db6bfeaaf9e1c88ab35. Latest consensus state: current_term: 1 leader_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } } }
I20260926 03:40:38.697962 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:38.697288 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } } }
I20260926 03:40:38.698155 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:38.699637 14340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:40:38.704668 14340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:40:38.706920 14329 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:40:38.721370 14340 catalog_manager.cc:1347] Generated new cluster ID: 59606d40ffa0426e82c77cbf7ea9d19a
I20260926 03:40:38.721560 14340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:40:38.740939 14340 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:40:38.742825 14340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:40:38.758194 14340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Generated new TSK 0
I20260926 03:40:38.759117 14340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:40:39.473939 14371 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:40:39.569203 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:40:39 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:40:39.966286 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:39.966787 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:39.966987 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:39.978807 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:39.979059 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:39.979310 14374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:40:39.995313 14374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:39.997246 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:39.998998 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:40.019025 14385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:40.020737 14387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:40.022496 14384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:40.023701 14374 server_base.cc:1034] running on GCE node
I20260926 03:40:40.024374 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:40.025561 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:40.026798 14374 hybrid_clock.cc:648] HybridClock initialized: now 1790394040026737 us; error 57 us; skew 500 ppm
I20260926 03:40:40.027129 14374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:40.030293 14374 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:40:40.032017 14374 webserver.cc:460] Webserver started at http://127.1.221.129:42963/ using document root <none> and password file <none>
I20260926 03:40:40.032862 14374 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:40.033002 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:40.033326 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:40.053738 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "31f69f2c64594b12bfe1b8c2db39d237"
format_stamp: "Formatted at 2026-09-26 03:40:40 on dist-test-slave-lqrk"
server_key: "3079a20fb5b64c8cc476a05df40a5902"
server_key_iv: "59a4eb8362a73718d7c7a65ca429ba62"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:40.055074 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "31f69f2c64594b12bfe1b8c2db39d237"
format_stamp: "Formatted at 2026-09-26 03:40:40 on dist-test-slave-lqrk"
server_key: "3079a20fb5b64c8cc476a05df40a5902"
server_key_iv: "59a4eb8362a73718d7c7a65ca429ba62"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:40.074239 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.005s sys 0.011s
I20260926 03:40:40.097246 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:40.099733 14374 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:40:40.099990 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "31f69f2c64594b12bfe1b8c2db39d237"
format_stamp: "Formatted at 2026-09-26 03:40:40 on dist-test-slave-lqrk"
server_key: "3079a20fb5b64c8cc476a05df40a5902"
server_key_iv: "59a4eb8362a73718d7c7a65ca429ba62"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:40.100325 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:40.120635 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:40.134354 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:40.134820 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:40.147893 14374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:40.160605 14374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:40.160816 14374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:40:40.161039 14374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:40.161165 14374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:40.245224 14374 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33823
I20260926 03:40:40.245285 14506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33823 every 8 connection(s)
I20260926 03:40:40.247404 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:40:40.255460 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14374
I20260926 03:40:40.255667 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:40:40.256487 1910 external_mini_cluster.cc:1428] Setting key 1a5388259f9c66a6ee5c8a77de207328
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:40.312537 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:40.251724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.129:51491)
0926 03:40:40.258267 (+ 6543us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:40.258357 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:40.274422 (+ 16065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:40.280539 (+ 6117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:40.280557 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:40.281363 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:40.282450 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:40.282484 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:40.284364 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:40.284374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:40.286716 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:40.286744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:40.297023 (+ 10279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:40.297033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:40.297498 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:40.298551 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:40.298599 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:40.301895 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:40.306637 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:40.306650 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:40.306677 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:40.307440 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:40.307970 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:40.307975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:40.307979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:40.308306 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:40.309119 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:40.309138 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:40.309712 (+ 574us) client_negotiation.cc:769] Sending connection context
0926 03:40:40.310084 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 03:40:40.310774 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6001,"thread_start_us":5385,"threads_started":1}
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:40.313957 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:40.254233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:51491 (local address 127.1.221.190:43715)
0926 03:40:40.279456 (+ 25223us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:40.279464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:40.279496 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:40:40.279586 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:40.279592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:40.279718 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:40.279879 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:40.282743 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:40.284035 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:40.295428 (+ 11393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:40.296264 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:40.296516 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:40.302328 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:40.302336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:40.302342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:40.302407 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:40.306411 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:40.307626 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:40.307632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:40.307636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:40.307824 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:40.308454 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:40.308459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:40.308463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:40.308888 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:40.309030 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:40.313234 (+ 4204us) server_negotiation.cc:299] Negotiation successful
0926 03:40:40.313533 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25004,"thread_start_us":24845,"threads_started":1}
I20260926 03:40:40.317606 14507 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:40.317998 14507 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:40.319020 14507 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:40.322275 14294 ts_manager.cc:194] Registered new tserver with Master: 31f69f2c64594b12bfe1b8c2db39d237 (127.1.221.129:33823)
I20260926 03:40:40.324752 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:51491
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
W20260926 03:40:40.370215 1910 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.221.130: real 0.110s user 0.001s sys 0.022s
I20260926 03:40:40.371340 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
W20260926 03:40:40.883978 14513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:40.884588 14513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:40.884786 14513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:40.896430 14513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:40.896700 14513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:40.896935 14513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:40:40.912820 14513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:40.915081 14513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:40.917071 14513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:40.955586 14521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:40.960456 14520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:40.968801 14513 server_base.cc:1034] running on GCE node
W20260926 03:40:40.969237 14524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:40.970189 14513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:40.979817 14513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:40.981523 14513 hybrid_clock.cc:648] HybridClock initialized: now 1790394040981393 us; error 144 us; skew 500 ppm
I20260926 03:40:40.982225 14513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:40 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:40.987164 14513 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:40:41.009269 14513 webserver.cc:460] Webserver started at http://127.1.221.130:42865/ using document root <none> and password file <none>
I20260926 03:40:41.010777 14513 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:41.011050 14513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:41.011607 14513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:40:41 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:41.034932 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "1c7ed05ace7148aaaff573c2409d548a"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "1d1c4ad02b061d1e0aed14866c5371f6"
server_key_iv: "37dc501f1a93c993c40d06d2282412cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.036468 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "1c7ed05ace7148aaaff573c2409d548a"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "1d1c4ad02b061d1e0aed14866c5371f6"
server_key_iv: "37dc501f1a93c993c40d06d2282412cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.074112 14513 fs_manager.cc:691] Time spent creating directory manager: real 0.037s user 0.017s sys 0.015s
I20260926 03:40:41.098711 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:41.100888 14513 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260926 03:40:41.101219 14513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "1c7ed05ace7148aaaff573c2409d548a"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "1d1c4ad02b061d1e0aed14866c5371f6"
server_key_iv: "37dc501f1a93c993c40d06d2282412cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.101588 14513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:41.160581 14513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:41.166190 14513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:41.166566 14513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:41.174036 14513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:41.182654 14513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:41.182852 14513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 03:40:41.183050 14513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:41.183137 14513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:41.329439 14507 heartbeater.cc:503] Master 127.1.221.190:43715 was elected leader, sending a full tablet report...
I20260926 03:40:41.452275 14513 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38983
I20260926 03:40:41.452342 14649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38983 every 8 connection(s)
I20260926 03:40:41.454216 14513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:40:41.459702 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14513
I20260926 03:40:41.459910 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:40:41.460448 1910 external_mini_cluster.cc:1428] Setting key 373660fa012c373420c73eac46795bdc
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Sep 26 03:40:41 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:41.507318 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:41.459872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41617 (local address 127.1.221.190:43715)
0926 03:40:41.481298 (+ 21426us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:41.481307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:41.485802 (+ 4495us) server_negotiation.cc:407] Connection header received
0926 03:40:41.485905 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:41.485910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:41.486020 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:41.486133 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:41.489813 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:41.491018 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:41.493497 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:41.494123 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:41.494236 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:41.499163 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:41.499172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:41.499176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:41.499240 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:41.502983 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:41.504097 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:41.504104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:41.504107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:41.504294 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:41.504844 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:41.504849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:41.504852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:41.505267 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:41.505405 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:41.506594 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 03:40:41.506880 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21203,"thread_start_us":21459,"threads_started":1}
I20260926 03:40:41.510452 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:41.459042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.130:41617)
0926 03:40:41.482237 (+ 23195us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:41.482329 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:41.485146 (+ 2817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:41.487816 (+ 2670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:41.487832 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:41.488579 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:41.489549 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:41.489579 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:41.491262 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:41.491267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:41.493308 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:41.493335 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:41.494643 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:41.494649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:41.494956 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:41.495925 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:41.495959 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:41.498823 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:41.503207 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:41.503218 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:41.503257 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:41.503945 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:41.504442 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:41.504446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:41.504450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:41.504713 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:41.505540 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:41.505549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:41.506099 (+ 550us) client_negotiation.cc:769] Sending connection context
0926 03:40:41.508033 (+ 1934us) client_negotiation.cc:240] Negotiation successful
0926 03:40:41.508728 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22720,"thread_start_us":27111,"threads_started":1}
I20260926 03:40:41.512919 14650 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:41.513299 14650 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:41.514358 14650 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:41.516564 14294 ts_manager.cc:194] Registered new tserver with Master: 1c7ed05ace7148aaaff573c2409d548a (127.1.221.130:38983)
I20260926 03:40:41.518355 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:41617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:40:41.531131 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:41.883323 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:41.883662 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:41.883803 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:41.891044 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:41.891165 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:41.891310 14656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:40:41.901261 14656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:41.902657 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:41.904018 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:41.917788 14664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:41.917929 14662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:41.917744 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:41.919070 14656 server_base.cc:1034] running on GCE node
I20260926 03:40:41.919760 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:41.920804 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:41.922073 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790394041922011 us; error 42 us; skew 500 ppm
I20260926 03:40:41.922457 14656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:41 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:41.926074 14656 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:40:41.927857 14656 webserver.cc:460] Webserver started at http://127.1.221.131:43165/ using document root <none> and password file <none>
I20260926 03:40:41.928609 14656 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:41.928740 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:41.929207 14656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:40:41 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:41.945894 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "15638add34b94ebe904e8ccb20c7073f"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "7fe2c37b1763755172f287b2eac20070"
server_key_iv: "198d0dfba6b0bf26c27f4db7d5467501"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.946923 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "15638add34b94ebe904e8ccb20c7073f"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "7fe2c37b1763755172f287b2eac20070"
server_key_iv: "198d0dfba6b0bf26c27f4db7d5467501"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.954159 14656 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 03:40:41.974500 14671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:41.976338 14656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:40:41.976536 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "15638add34b94ebe904e8ccb20c7073f"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "7fe2c37b1763755172f287b2eac20070"
server_key_iv: "198d0dfba6b0bf26c27f4db7d5467501"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:41.976780 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:41.996675 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:42.000680 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:42.000965 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:42.002187 14656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:42.003768 14656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:42.003844 14656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:42.003950 14656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:42.004014 14656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:42.039011 14656 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:40115
I20260926 03:40:42.039076 14783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:40115 every 8 connection(s)
I20260926 03:40:42.040386 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:40:42.041781 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14656
I20260926 03:40:42.041985 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:40:42.042631 1910 external_mini_cluster.cc:1428] Setting key 55c8e9513d495f7b58d8ad98c0e82a5a
Sep 26 03:40:42 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:42.064340 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:42.043864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60353 (local address 127.1.221.190:43715)
0926 03:40:42.044537 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:42.044542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:42.045949 (+ 1407us) server_negotiation.cc:407] Connection header received
0926 03:40:42.048370 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:42.048375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:42.048469 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:42.048597 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:42.050301 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.051091 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.053111 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.053541 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.053654 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:42.057461 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:42.057469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:42.057474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:42.057530 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:42.060805 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.061747 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.061751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.061754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.061896 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.062438 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.062442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.062444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.062745 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:42.062869 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:42.063833 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 03:40:42.064054 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":336,"threads_started":1}
I20260926 03:40:42.065631 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:42.044677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.131:60353)
0926 03:40:42.045706 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:42.045796 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:42.048053 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:42.048840 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:42.048882 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:42.049378 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:42.050111 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:42.050130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.051262 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.051267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:42.052933 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:42.052947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.053763 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.053768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:42.054042 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:42.054760 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:42.054785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:42.057179 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:42.060975 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:42.060994 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:42.061016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:42.061583 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:42.062039 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:42.062043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:42.062046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:42.062304 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:42.062872 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:42.062880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:42.063349 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 03:40:42.063691 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 03:40:42.064278 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":399,"threads_started":1}
I20260926 03:40:42.067606 14784 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:42.067950 14784 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:42.068807 14784 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:42.070746 14294 ts_manager.cc:194] Registered new tserver with Master: 15638add34b94ebe904e8ccb20c7073f (127.1.221.131:40115)
I20260926 03:40:42.072124 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:60353
I20260926 03:40:42.073498 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:40:42.075618 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14374
Sep 26 03:40:42 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:40:42.094184 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:42.076712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.1.221.129:33823)
0926 03:40:42.077382 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:42.077391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:42.077426 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:40:42.077501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:42.077509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:42.077773 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:42.077953 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:42.079601 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.080807 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.082498 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.083181 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.083305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:42.087599 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:42.087613 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:42.087631 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:42.087689 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:42.091014 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.091914 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.091923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.091926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.092110 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.092510 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.092514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.092516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.092939 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:42.093222 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:42.093570 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:40:42.093858 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":302,"threads_started":1}
I20260926 03:40:42.095854 14502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:42.181741 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14374
I20260926 03:40:42.194921 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14513
Sep 26 03:40:42 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:40:42.213964 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:42.196132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.1.221.130:38983)
0926 03:40:42.196851 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:42.196861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:42.196895 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:40:42.196975 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:42.196983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:42.197277 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:42.197445 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:42.198675 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.199980 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.201393 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.202164 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.202296 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:42.206955 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:42.206973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:42.206998 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:42.207064 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:42.210554 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.211468 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.211478 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.211480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.211664 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.212050 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.212056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.212059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.212529 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:42.212800 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:42.213148 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:40:42.213454 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":385,"threads_started":1}
I20260926 03:40:42.215571 14645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:42.298992 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14513
I20260926 03:40:42.315192 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14656
Sep 26 03:40:42 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:40:42.335074 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:42.316530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.1.221.131:40115)
0926 03:40:42.317420 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:42.317429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:42.317469 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:40:42.317549 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:42.317558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:42.317857 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:42.318031 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:42.319725 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.320883 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.322520 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:42.323221 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:42.323368 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:42.328715 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:42.328735 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:42.328755 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:42.328804 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:42.331876 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.332873 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.332881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.332884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.333083 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:42.333522 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:42.333525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:42.333527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:42.333915 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:42.334223 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:42.334492 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:40:42.334732 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":469,"threads_started":1}
I20260926 03:40:42.336936 14779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:42.427729 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14656
I20260926 03:40:42.443010 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14262
I20260926 03:40:42.444666 14324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:42.553063 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14262
I20260926 03:40:42.574234 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43715
--webserver_interface=127.1.221.190
--webserver_port=45121
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43715
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:42.909543 14796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:42.909871 14796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:42.909974 14796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:42.917912 14796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:42.918056 14796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:42.918102 14796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:42.918166 14796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:42.918242 14796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:42.928817 14796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43715
--ranger_config_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=45121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:42.930217 14796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:42.931582 14796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:42.944669 14801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:42.944701 14802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:42.944770 14804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:42.945801 14796 server_base.cc:1034] running on GCE node
I20260926 03:40:42.946655 14796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:42.948302 14796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:42.949540 14796 hybrid_clock.cc:648] HybridClock initialized: now 1790394042949478 us; error 65 us; skew 500 ppm
I20260926 03:40:42.949913 14796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:42 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:42.953570 14796 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:40:42.955291 14796 webserver.cc:460] Webserver started at http://127.1.221.190:45121/ using document root <none> and password file <none>
I20260926 03:40:42.955998 14796 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:42.956138 14796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:40:43 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:43.023351 14796 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 03:40:43.027046 14812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:43.028636 14796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 03:40:43.028810 14796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "c03fe24467d94db6bfeaaf9e1c88ab35"
format_stamp: "Formatted at 2026-09-26 03:40:38 on dist-test-slave-lqrk"
server_key: "973ef730e5bb86cbe43e5cd166d9a386"
server_key_iv: "8947ab65629bd5bd16ba4a92bf293a0d"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:43.029798 14796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:43.062930 14796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:43.067157 14796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:43.067632 14796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:43.087751 14796 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43715
I20260926 03:40:43.087803 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43715 every 8 connection(s)
I20260926 03:40:43.089368 14796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:40:43.093125 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14796
I20260926 03:40:43.093988 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:33823
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:43715
--webserver_port=42963
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:40:43.098376 14864 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:40:43.102550 14864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Bootstrap starting.
I20260926 03:40:43.126094 14864 log.cc:824] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:43.137878 14864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:40:43.138571 14864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Bootstrap complete.
I20260926 03:40:43.148468 14864 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:43.149609 14864 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03fe24467d94db6bfeaaf9e1c88ab35, State: Initialized, Role: FOLLOWER
I20260926 03:40:43.150450 14864 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:43.150743 14864 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:40:43.150878 14864 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:40:43.151067 14864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:40:43.155668 14864 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:43.156280 14864 leader_election.cc:302] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03fe24467d94db6bfeaaf9e1c88ab35; no voters:
I20260926 03:40:43.156953 14864 leader_election.cc:288] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:40:43.157119 14868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:40:43.158350 14868 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [term 2 LEADER]: Becoming Leader. State: Replica: c03fe24467d94db6bfeaaf9e1c88ab35, State: Running, Role: LEADER
I20260926 03:40:43.158998 14868 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } }
I20260926 03:40:43.159619 14864 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:43.162153 14869 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } } }
I20260926 03:40:43.162626 14869 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:43.167361 14870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03fe24467d94db6bfeaaf9e1c88ab35. Latest consensus state: current_term: 2 leader_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03fe24467d94db6bfeaaf9e1c88ab35" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43715 } } }
I20260926 03:40:43.167754 14870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:43.169508 14873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:40:43.177487 14873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:40:43.179111 14873 catalog_manager.cc:1259] Loaded cluster ID: 59606d40ffa0426e82c77cbf7ea9d19a
I20260926 03:40:43.179224 14873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:40:43.184309 14873 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:40:43.186019 14873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Loaded TSK: 0
I20260926 03:40:43.188043 14873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:40:43.192220 14864 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 03:40:43.575207 14866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:43.575572 14866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:43.575731 14866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:43.586243 14866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:43.586398 14866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:43.586588 14866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:40:43.602521 14866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:33823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=42963
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:43.603996 14866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:43.605334 14866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:43.637326 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:43.638408 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:43.647454 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:43.648962 14866 server_base.cc:1034] running on GCE node
I20260926 03:40:43.650126 14866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:43.651816 14866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:43.659508 14866 hybrid_clock.cc:648] HybridClock initialized: now 1790394043659368 us; error 107 us; skew 500 ppm
I20260926 03:40:43.660159 14866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:43 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:43.669186 14866 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:40:43.684681 14866 webserver.cc:460] Webserver started at http://127.1.221.129:42963/ using document root <none> and password file <none>
I20260926 03:40:43.685725 14866 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:43.685899 14866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:40:43 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:43.736639 14866 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.007s sys 0.004s
I20260926 03:40:43.741479 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:43.743733 14866 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:40:43.743971 14866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "31f69f2c64594b12bfe1b8c2db39d237"
format_stamp: "Formatted at 2026-09-26 03:40:40 on dist-test-slave-lqrk"
server_key: "3079a20fb5b64c8cc476a05df40a5902"
server_key_iv: "59a4eb8362a73718d7c7a65ca429ba62"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:43.745251 14866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:43.770458 14866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:43.775655 14866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:43.775998 14866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:43.777297 14866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:43.779155 14866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:43.779263 14866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:43.779409 14866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:43.779490 14866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:40:43.836093 14866 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33823
I20260926 03:40:43.836280 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33823 every 8 connection(s)
I20260926 03:40:43.838069 14866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:40:43.839416 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 14866
I20260926 03:40:43.840387 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:38983
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:43715
--webserver_port=42865
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:40:43 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394043, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:43.891606 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:43.843765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.129:60439)
0926 03:40:43.854094 (+ 10329us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:43.854181 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:43.857065 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:43.868402 (+ 11337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:43.868418 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:43.869273 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:43.870274 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:43.870313 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:43.873513 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:43.873520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:43.875835 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:43.875856 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:43.878000 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:43.878006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:43.878225 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:43.879103 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:43.879136 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:43.881688 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:43.885751 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:43.885762 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:43.885783 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:43.886281 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:43.886788 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:43.886793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:43.886797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:43.887094 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:43.888278 (+ 1184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:43.888300 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:43.888792 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 03:40:43.889154 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 03:40:43.889875 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9877,"thread_start_us":15499,"threads_started":1}
I20260926 03:40:43.894692 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:43.843615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60439 (local address 127.1.221.190:43715)
0926 03:40:43.857750 (+ 14135us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:43.857781 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:43.857870 (+ 89us) server_negotiation.cc:407] Connection header received
0926 03:40:43.865702 (+ 7832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:43.865736 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:43.866646 (+ 910us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:43.867199 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:43.870971 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:43.873219 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:43.876041 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:43.877287 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:43.877484 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:43.882064 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:43.882074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:43.882094 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:43.882142 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:43.885527 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:43.886470 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:43.886478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:43.886484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:43.886661 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:43.887247 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:43.887253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:43.887256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:43.887940 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:43.888185 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:43.892684 (+ 4499us) server_negotiation.cc:299] Negotiation successful
0926 03:40:43.893222 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13771,"thread_start_us":13471,"threads_started":1}
I20260926 03:40:43.899078 15026 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:43.899569 15026 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:43.900457 15026 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
W20260926 03:40:43.903924 15026 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:43715 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:40:43.904163 15026 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:43.904452 15026 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:43.905917 15026 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:43.906188 15026 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
W20260926 03:40:43.907827 15026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:40:44.241194 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:44.241530 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:44.241703 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:44.249516 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:44.249663 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:44.249814 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:40:44.260629 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:38983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=42865
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:44.262030 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:44.263293 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:44.277540 15042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:44.277710 15045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:44.277992 15043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:44.278539 15030 server_base.cc:1034] running on GCE node
I20260926 03:40:44.279222 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:44.280424 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:44.281661 15030 hybrid_clock.cc:648] HybridClock initialized: now 1790394044281606 us; error 46 us; skew 500 ppm
I20260926 03:40:44.281997 15030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:44 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:44.285722 15030 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:40:44.287781 15030 webserver.cc:460] Webserver started at http://127.1.221.130:42865/ using document root <none> and password file <none>
I20260926 03:40:44.288539 15030 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:44.288674 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:40:44 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:44.316800 15030 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 03:40:44.319888 15055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:44.321862 15030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:40:44.322142 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "1c7ed05ace7148aaaff573c2409d548a"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "1d1c4ad02b061d1e0aed14866c5371f6"
server_key_iv: "37dc501f1a93c993c40d06d2282412cc"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:44.325049 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:44.343847 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:44.349380 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:44.349709 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:44.350999 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:44.352885 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:44.353006 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:44.353120 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:44.353199 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:44.456950 15030 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38983
I20260926 03:40:44.457022 15169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38983 every 8 connection(s)
I20260926 03:40:44.458853 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:40:44.462306 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15030
I20260926 03:40:44.463121 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:47023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:40115
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:43715
--webserver_port=43165
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:40:44 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:44.498893 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:44.463294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39479 (local address 127.1.221.190:43715)
0926 03:40:44.473554 (+ 10260us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:44.473561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:44.474730 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 03:40:44.477350 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:44.477357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:44.477463 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:44.477573 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:44.480016 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:44.481287 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:44.483869 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:44.484573 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:44.484678 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:44.489562 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:44.489571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:44.489576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:44.489622 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:44.493693 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:44.494816 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:44.494823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:44.494827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:44.494982 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:44.495641 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:44.495647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:44.495650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:44.496036 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:44.496178 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:44.497765 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0926 03:40:44.498027 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10101,"thread_start_us":9964,"threads_started":1}
I20260926 03:40:44.502718 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:44.462597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.130:39479)
0926 03:40:44.474169 (+ 11572us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:44.474234 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:44.476790 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:44.478077 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:44.478091 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:44.478804 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:44.479733 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:44.479771 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:44.481515 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:44.481520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:44.483672 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:44.483690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:44.485089 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:44.485096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:44.485416 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:44.486290 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:44.486327 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:44.489219 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:44.493915 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:44.493926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:44.493947 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:44.494656 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:44.495142 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:44.495147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:44.495151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:44.495500 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:44.496275 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:44.496296 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:44.497295 (+ 999us) client_negotiation.cc:769] Sending connection context
0926 03:40:44.499506 (+ 2211us) client_negotiation.cc:240] Negotiation successful
0926 03:40:44.501192 (+ 1686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11238,"thread_start_us":11006,"threads_started":1}
I20260926 03:40:44.504670 15170 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:44.505002 15170 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:44.505895 15170 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:44.508199 14829 ts_manager.cc:194] Registered new tserver with Master: 1c7ed05ace7148aaaff573c2409d548a (127.1.221.130:38983)
I20260926 03:40:44.510531 14829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:39479
Sep 26 03:40:44 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:44.907994 15026 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:44.908484 15026 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:44.909807 14829 ts_manager.cc:194] Registered new tserver with Master: 31f69f2c64594b12bfe1b8c2db39d237 (127.1.221.129:33823)
I20260926 03:40:44.911060 14829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:60439
W20260926 03:40:44.939998 15174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:44.940393 15174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:44.940543 15174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:44.953230 15174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:44.953398 15174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:44.953580 15174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:40:44.970665 15174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:47023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:40115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=43165
--tserver_master_addrs=127.1.221.190:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:44.972611 15174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:44.974424 15174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:44.994844 15180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:44.995184 15181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 26 03:40:44 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
W20260926 03:40:44.999574 15183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:45.004938 15174 server_base.cc:1034] running on GCE node
I20260926 03:40:45.005944 15174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:45.019769 15174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:45.021201 15174 hybrid_clock.cc:648] HybridClock initialized: now 1790394045021115 us; error 87 us; skew 500 ppm
I20260926 03:40:45.021677 15174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:45 dist-test-slave-lqrk krb5kdc[13818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394045, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:45.026659 15174 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:40:45.033763 15174 webserver.cc:460] Webserver started at http://127.1.221.131:43165/ using document root <none> and password file <none>
I20260926 03:40:45.034834 15174 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:45.035071 15174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:40:45 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394045, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:45.098958 15174 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.002s sys 0.004s
I20260926 03:40:45.122718 15191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:45.125041 15174 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.002s sys 0.006s
I20260926 03:40:45.125305 15174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "15638add34b94ebe904e8ccb20c7073f"
format_stamp: "Formatted at 2026-09-26 03:40:41 on dist-test-slave-lqrk"
server_key: "7fe2c37b1763755172f287b2eac20070"
server_key_iv: "198d0dfba6b0bf26c27f4db7d5467501"
server_key_version: "kuduclusterkey@0"
I20260926 03:40:45.126890 15174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:45.171352 15174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:45.176791 15174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:45.177147 15174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:45.184671 15174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:45.193011 15174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:45.193233 15174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:40:45.193356 15174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:45.193431 15174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:40:45 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260926 03:40:45.516518 15170 heartbeater.cc:503] Master 127.1.221.190:43715 was elected leader, sending a full tablet report...
I20260926 03:40:45.623045 15174 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:40115
I20260926 03:40:45.623075 15304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:40115 every 8 connection(s)
I20260926 03:40:45.625025 15174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:40:45.629194 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15174
Sep 26 03:40:45 dist-test-slave-lqrk krb5kdc[13818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394045, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:45.658118 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.630645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.1.221.190:43715)
0926 03:40:45.636248 (+ 5603us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.636254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.638302 (+ 2048us) server_negotiation.cc:407] Connection header received
0926 03:40:45.638376 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.638381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.638575 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.638681 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:45.640946 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.642018 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.644000 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.644622 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.644720 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.648901 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:45.648909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:45.648914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:45.648965 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:45.652884 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.653870 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.653880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.653883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.654062 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.654615 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.654621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.654624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.655093 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:45.655422 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.657409 (+ 1987us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.657677 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5492,"thread_start_us":4977,"threads_started":1}
I20260926 03:40:45.659240 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.630136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:34923 (local address 127.1.221.190:43715)
0926 03:40:45.635892 (+ 5756us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.635899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.637466 (+ 1567us) server_negotiation.cc:407] Connection header received
0926 03:40:45.639665 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.639671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.639759 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.639837 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:45.642242 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.643445 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.645965 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.646600 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.646715 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.651840 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:45.651848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:45.651853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:45.651913 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:45.655153 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.656411 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.656416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.656419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.656544 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.657021 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.657026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.657029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.657391 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:45.657540 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.658601 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.658834 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5599,"thread_start_us":5858,"threads_started":1}
I20260926 03:40:45.663105 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.629341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43715 (local address 127.1.221.131:34923)
0926 03:40:45.636812 (+ 7471us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.636873 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.639274 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.640483 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.640496 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.641162 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:45.642030 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.642057 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.643666 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.643672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.645783 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.645801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.647143 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.647152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.647505 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.648258 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:45.648297 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:45.651472 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:45.655366 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:45.655408 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:45.655431 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:45.656274 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:45.656668 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:45.656672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:45.656674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:45.656895 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:45.657621 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:45.657629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:45.658121 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.660168 (+ 2047us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.660661 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7158,"thread_start_us":6917,"threads_started":1}
I20260926 03:40:45.665136 15305 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43715
I20260926 03:40:45.665483 15305 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:45.666167 15305 heartbeater.cc:511] Master 127.1.221.190:43715 requested a full tablet report, sending...
I20260926 03:40:45.667769 14829 ts_manager.cc:194] Registered new tserver with Master: 15638add34b94ebe904e8ccb20c7073f (127.1.221.131:40115)
I20260926 03:40:45.668710 14829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:34923
I20260926 03:40:45.674803 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:40:45.716384 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.689986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.1.221.190:43715)
0926 03:40:45.690181 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.690188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.700115 (+ 9927us) server_negotiation.cc:407] Connection header received
0926 03:40:45.700315 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.700321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.700435 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.700544 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:45.702495 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.703443 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.705186 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.705735 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.705866 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.709375 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:45.709382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:45.709387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:45.709437 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:45.712903 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.713896 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.713903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.713906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.714074 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:45.714553 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:45.714559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:45.714562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:45.714937 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:45.715160 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.715783 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.716031 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:40:45.722805 14829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:40:45.733275 14829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:40:45.803066 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.752863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40690)
0926 03:40:45.791234 (+ 38371us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.791257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.791421 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.793183 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.793191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.793225 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:45.793608 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.793621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.797046 (+ 3425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.797052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.799604 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.799619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.801997 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.802004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.802229 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.802281 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.802441 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.802670 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38232,"thread_start_us":38646,"threads_started":1}
I20260926 03:40:45.804777 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.753126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.1.221.129:33823)
0926 03:40:45.792177 (+ 39051us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.792187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.792221 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:40:45.792336 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.792345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.792650 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.792867 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:45.795087 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.796857 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.799869 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.801437 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.801571 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.801824 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.804250 (+ 2426us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.804455 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38834,"thread_start_us":38988,"threads_started":1}
I20260926 03:40:45.809069 14961 tablet_service.cc:1467] Processing CreateTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 (DEFAULT_TABLE table=test-table [id=04b6c80ff7cd4f2d99d05dd88e25ffd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:45.811113 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.752614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:40115 (local address 127.0.0.1:33522)
0926 03:40:45.788870 (+ 36256us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.788923 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.789121 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.793907 (+ 4786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.793912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.793969 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:45.794548 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.794564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.804916 (+ 10352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.804921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.808204 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.808226 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.810205 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.810213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.810412 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.810451 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.810565 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.810735 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35918,"thread_start_us":51982,"threads_started":1}
I20260926 03:40:45.811897 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.753316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.1.221.131:40115)
0926 03:40:45.793004 (+ 39688us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.793013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.793046 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:40:45.793141 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.793149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.793376 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.794901 (+ 1525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:45.795262 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.796855 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.808504 (+ 11649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.809921 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.810078 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.810322 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.811370 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.811569 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39518,"thread_start_us":40151,"threads_started":1}
I20260926 03:40:45.812753 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6bb6c537c74ba3849678eeffa3c301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:45.813308 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.756252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38983 (local address 127.0.0.1:34556)
0926 03:40:45.799046 (+ 42794us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.799067 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.799212 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.800271 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.800276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.800304 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:45.800638 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.800664 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.802536 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.802540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.804487 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.804503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.812511 (+ 8008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.812518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.812693 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.812728 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.812834 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.812991 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42646,"thread_start_us":33198,"threads_started":1}
I20260926 03:40:45.813462 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.756620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.1.221.130:38983)
0926 03:40:45.790637 (+ 34017us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.790646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.799193 (+ 8547us) server_negotiation.cc:407] Connection header received
0926 03:40:45.799426 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.799436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.799789 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.799989 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:45.800854 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.802364 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.810461 (+ 8097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.811942 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.812064 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.812309 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.812917 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.813102 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33820,"thread_start_us":34040,"threads_started":1}
I20260926 03:40:45.818202 15240 tablet_service.cc:1467] Processing CreateTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 (DEFAULT_TABLE table=test-table [id=04b6c80ff7cd4f2d99d05dd88e25ffd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:45.818184 15105 tablet_service.cc:1467] Processing CreateTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 (DEFAULT_TABLE table=test-table [id=04b6c80ff7cd4f2d99d05dd88e25ffd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:45.819792 15240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6bb6c537c74ba3849678eeffa3c301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:45.819835 15105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6bb6c537c74ba3849678eeffa3c301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:45.852160 15328 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: Bootstrap starting.
I20260926 03:40:45.854076 15330 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Bootstrap starting.
I20260926 03:40:45.856271 15331 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Bootstrap starting.
I20260926 03:40:45.860177 15330 tablet_bootstrap.cc:654] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:45.861423 15328 tablet_bootstrap.cc:654] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:45.861635 15330 log.cc:824] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:45.863123 15331 tablet_bootstrap.cc:654] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:45.863278 15328 log.cc:824] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:45.864419 15331 log.cc:824] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:45.865196 15330 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: No bootstrap required, opened a new log
I20260926 03:40:45.865582 15330 ts_tablet_manager.cc:1397] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260926 03:40:45.874501 15331 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: No bootstrap required, opened a new log
I20260926 03:40:45.875449 15331 ts_tablet_manager.cc:1397] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260926 03:40:45.876778 15330 raft_consensus.cc:348] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.877269 15330 raft_consensus.cc:374] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:45.877378 15330 raft_consensus.cc:729] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed05ace7148aaaff573c2409d548a, State: Initialized, Role: FOLLOWER
I20260926 03:40:45.879155 15330 consensus_queue.cc:260] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.880882 15328 tablet_bootstrap.cc:492] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: No bootstrap required, opened a new log
I20260926 03:40:45.881199 15328 ts_tablet_manager.cc:1397] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.008s
I20260926 03:40:45.887465 15331 raft_consensus.cc:348] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.888109 15331 raft_consensus.cc:374] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:45.888254 15331 raft_consensus.cc:729] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15638add34b94ebe904e8ccb20c7073f, State: Initialized, Role: FOLLOWER
I20260926 03:40:45.889191 15330 ts_tablet_manager.cc:1428] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20260926 03:40:45.889092 15331 consensus_queue.cc:260] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.891860 15328 raft_consensus.cc:348] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.892331 15328 raft_consensus.cc:374] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:45.892464 15328 raft_consensus.cc:729] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f69f2c64594b12bfe1b8c2db39d237, State: Initialized, Role: FOLLOWER
I20260926 03:40:45.893263 15328 consensus_queue.cc:260] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.897733 15331 ts_tablet_manager.cc:1428] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Time spent starting tablet: real 0.022s user 0.016s sys 0.003s
I20260926 03:40:45.898159 15305 heartbeater.cc:503] Master 127.1.221.190:43715 was elected leader, sending a full tablet report...
I20260926 03:40:45.898737 15026 heartbeater.cc:503] Master 127.1.221.190:43715 was elected leader, sending a full tablet report...
I20260926 03:40:45.899443 15328 ts_tablet_manager.cc:1428] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
I20260926 03:40:45.945657 15340 raft_consensus.cc:482] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:40:45.945941 15340 raft_consensus.cc:504] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.947706 15340 leader_election.cc:288] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f69f2c64594b12bfe1b8c2db39d237 (127.1.221.129:33823), 1c7ed05ace7148aaaff573c2409d548a (127.1.221.130:38983)
I20260926 03:40:45.955166 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.947954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.1.221.131:58829)
0926 03:40:45.948166 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.948185 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.948333 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.949023 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.949028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.949057 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:45.949303 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.949319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.950805 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.950811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.952876 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.952891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.954411 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.954417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.954608 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.954646 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.954745 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.954885 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:40:45.955169 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.948406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:58829 (local address 127.1.221.129:33823)
0926 03:40:45.948585 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.948591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.948609 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:40:45.948684 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.948689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.948763 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.948847 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:45.949554 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.950653 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.953043 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.954251 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.954424 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.954499 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.954802 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.954929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:40:45.955451 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.948827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35365 (local address 127.1.221.130:38983)
0926 03:40:45.949028 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:45.949034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:45.949609 (+ 575us) server_negotiation.cc:407] Connection header received
0926 03:40:45.949750 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:45.949754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:45.949813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:45.949884 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:45.950409 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.951478 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.953574 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.954357 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.954502 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.954576 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:45.955023 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 03:40:45.955166 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:40:45.955978 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:45.948324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38983 (local address 127.1.221.131:35365)
0926 03:40:45.949512 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:45.949526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:45.949630 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:45.949909 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:45.949913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:45.949937 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:45.950262 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.950272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.951690 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.951699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.953430 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:45.953440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:45.954654 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:45.954660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:45.954840 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:45.954872 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:40:45.955532 (+ 660us) client_negotiation.cc:240] Negotiation successful
0926 03:40:45.955663 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":904,"threads_started":1}
I20260926 03:40:45.956636 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6bb6c537c74ba3849678eeffa3c301" candidate_uuid: "15638add34b94ebe904e8ccb20c7073f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f69f2c64594b12bfe1b8c2db39d237" is_pre_election: true
I20260926 03:40:45.957168 14981 raft_consensus.cc:2454] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15638add34b94ebe904e8ccb20c7073f in term 0.
I20260926 03:40:45.957077 15125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6bb6c537c74ba3849678eeffa3c301" candidate_uuid: "15638add34b94ebe904e8ccb20c7073f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed05ace7148aaaff573c2409d548a" is_pre_election: true
I20260926 03:40:45.957587 15125 raft_consensus.cc:2454] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15638add34b94ebe904e8ccb20c7073f in term 0.
I20260926 03:40:45.957815 15192 leader_election.cc:302] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15638add34b94ebe904e8ccb20c7073f, 31f69f2c64594b12bfe1b8c2db39d237; no voters:
I20260926 03:40:45.958236 15340 raft_consensus.cc:2781] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:40:45.958360 15340 raft_consensus.cc:482] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:40:45.958461 15340 raft_consensus.cc:3037] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:45.960840 15340 raft_consensus.cc:504] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.961436 15340 leader_election.cc:288] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [CANDIDATE]: Term 1 election: Requested vote from peers 31f69f2c64594b12bfe1b8c2db39d237 (127.1.221.129:33823), 1c7ed05ace7148aaaff573c2409d548a (127.1.221.130:38983)
I20260926 03:40:45.961988 15125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6bb6c537c74ba3849678eeffa3c301" candidate_uuid: "15638add34b94ebe904e8ccb20c7073f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed05ace7148aaaff573c2409d548a"
I20260926 03:40:45.962044 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6bb6c537c74ba3849678eeffa3c301" candidate_uuid: "15638add34b94ebe904e8ccb20c7073f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f69f2c64594b12bfe1b8c2db39d237"
I20260926 03:40:45.962226 15125 raft_consensus.cc:3037] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:45.962232 14981 raft_consensus.cc:3037] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:45.965435 15125 raft_consensus.cc:2454] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15638add34b94ebe904e8ccb20c7073f in term 1.
I20260926 03:40:45.965435 14981 raft_consensus.cc:2454] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15638add34b94ebe904e8ccb20c7073f in term 1.
I20260926 03:40:45.966019 15192 leader_election.cc:302] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15638add34b94ebe904e8ccb20c7073f, 31f69f2c64594b12bfe1b8c2db39d237; no voters:
I20260926 03:40:45.966368 15340 raft_consensus.cc:2781] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:45.966885 15340 raft_consensus.cc:686] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 1 LEADER]: Becoming Leader. State: Replica: 15638add34b94ebe904e8ccb20c7073f, State: Running, Role: LEADER
W20260926 03:40:45.967648 15171 tablet.cc:2369] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:45.967666 15340 consensus_queue.cc:237] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } }
I20260926 03:40:45.973053 14829 catalog_manager.cc:5495] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f reported cstate change: term changed from 0 to 1, leader changed from <none> to 15638add34b94ebe904e8ccb20c7073f (127.1.221.131). New cstate: current_term: 1 leader_uuid: "15638add34b94ebe904e8ccb20c7073f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15638add34b94ebe904e8ccb20c7073f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 } health_report { overall_health: UNKNOWN } } }
I20260926 03:40:46.011981 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.006760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.1.221.131:40115)
0926 03:40:46.006984 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:46.006990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:46.007009 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:40:46.007180 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:46.007186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:46.007265 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:46.007418 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:40:46.007789 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.008489 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.009839 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.010238 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.010345 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:46.010858 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:40:46.011372 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:40:46.011537 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:46.011636 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:40:46.011736 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:40:46.021579 15125 raft_consensus.cc:1261] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 1 FOLLOWER]: Refusing update from remote peer 15638add34b94ebe904e8ccb20c7073f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:46.021612 14981 raft_consensus.cc:1261] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 1 FOLLOWER]: Refusing update from remote peer 15638add34b94ebe904e8ccb20c7073f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:46.022607 15340 consensus_queue.cc:1035] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f69f2c64594b12bfe1b8c2db39d237" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:46.022955 15343 consensus_queue.cc:1035] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7ed05ace7148aaaff573c2409d548a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:46.040026 15348 mvcc.cc:205] Tried to move back new op lower bound from 7333454012492955648 to 7333454012288487424. Current Snapshot: MvccSnapshot[applied={T|T < 7333454012492955648}]
I20260926 03:40:46.055473 14829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:40:46.055790 14829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:40:46.060636 14829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c03fe24467d94db6bfeaaf9e1c88ab35: Sending DeleteTablet for 3 replicas of tablet ba6bb6c537c74ba3849678eeffa3c301
I20260926 03:40:46.061986 15240 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33522
I20260926 03:40:46.063511 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40690
I20260926 03:40:46.070987 15105 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6bb6c537c74ba3849678eeffa3c301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34556
I20260926 03:40:46.075685 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14866
I20260926 03:40:46.090169 15357 tablet_replica.cc:330] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: stopping tablet replica
I20260926 03:40:46.090766 15357 raft_consensus.cc:2229] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:40:46.091372 15357 raft_consensus.cc:2258] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:46.112112 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.076891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.1.221.129:33823)
0926 03:40:46.077078 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:46.077083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:46.077102 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:40:46.077178 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:46.077184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:46.077265 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:46.077368 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:46.088733 (+ 11365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.089977 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.093360 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.094058 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.094200 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:46.102910 (+ 8710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:46.102934 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:46.102955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:46.103012 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:46.107210 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.108441 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.108453 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.108461 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.108659 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.109150 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.109156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.109160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.109693 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:46.110019 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:46.111446 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0926 03:40:46.111697 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:40:46.112839 15021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:46.127509 15355 tablet_replica.cc:330] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: stopping tablet replica
I20260926 03:40:46.127995 15355 raft_consensus.cc:2229] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:40:46.128757 15355 raft_consensus.cc:2258] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:46.136394 15357 ts_tablet_manager.cc:1905] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:40:46.142040 15355 ts_tablet_manager.cc:1905] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:40:46.151715 15357 ts_tablet_manager.cc:1918] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:40:46.152019 15357 log.cc:1196] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/ba6bb6c537c74ba3849678eeffa3c301
I20260926 03:40:46.152597 15357 ts_tablet_manager.cc:1939] T ba6bb6c537c74ba3849678eeffa3c301 P 1c7ed05ace7148aaaff573c2409d548a: Deleting consensus metadata
I20260926 03:40:46.153710 15355 ts_tablet_manager.cc:1918] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:40:46.153908 15355 log.cc:1196] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/ba6bb6c537c74ba3849678eeffa3c301
I20260926 03:40:46.154735 15355 ts_tablet_manager.cc:1939] T ba6bb6c537c74ba3849678eeffa3c301 P 15638add34b94ebe904e8ccb20c7073f: Deleting consensus metadata
I20260926 03:40:46.154757 14814 catalog_manager.cc:4841] TS 1c7ed05ace7148aaaff573c2409d548a (127.1.221.130:38983): tablet ba6bb6c537c74ba3849678eeffa3c301 (table test-table [id=04b6c80ff7cd4f2d99d05dd88e25ffd4]) successfully deleted
I20260926 03:40:46.156572 14816 catalog_manager.cc:4841] TS 15638add34b94ebe904e8ccb20c7073f (127.1.221.131:40115): tablet ba6bb6c537c74ba3849678eeffa3c301 (table test-table [id=04b6c80ff7cd4f2d99d05dd88e25ffd4]) successfully deleted
I20260926 03:40:46.293062 15356 tablet_replica.cc:330] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237: stopping tablet replica
I20260926 03:40:46.293579 15356 raft_consensus.cc:2229] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:40:46.294026 15356 raft_consensus.cc:2258] T ba6bb6c537c74ba3849678eeffa3c301 P 31f69f2c64594b12bfe1b8c2db39d237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:46.294130 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14866
I20260926 03:40:46.307946 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15030
W20260926 03:40:46.307865 14813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:33823 (error 108)
I20260926 03:40:46.309719 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.309275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40708)
0926 03:40:46.309449 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.309572 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:40:46.311300 14813 catalog_manager.cc:4584] TS 31f69f2c64594b12bfe1b8c2db39d237 (127.1.221.129:33823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba6bb6c537c74ba3849678eeffa3c301: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
I20260926 03:40:46.329252 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.309006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.1.221.130:38983)
0926 03:40:46.309157 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:46.309161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:46.309177 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:40:46.309238 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:46.309241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:46.309304 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:46.309404 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:46.311254 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.312209 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.314141 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.314577 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.314673 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:46.323134 (+ 8461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:46.323156 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:46.323173 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:46.323217 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:46.326130 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.327037 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.327045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.327047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.327203 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.327731 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.327735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.327737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.328101 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:46.328344 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:46.328752 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:40:46.328935 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:40:46.329955 15165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:46.365420 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.364851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40710)
0926 03:40:46.365103 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.365240 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:40:46.366627 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.366213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40720)
0926 03:40:46.366379 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.366477 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:40:46.421469 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15030
I20260926 03:40:46.427802 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.426986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40724)
0926 03:40:46.427295 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.427464 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 03:40:46.430462 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.429887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40726)
0926 03:40:46.430131 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.430265 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:40:46.438287 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15174
I20260926 03:40:46.454559 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.438992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.1.221.131:40115)
0926 03:40:46.439152 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:46.439156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:46.439257 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:40:46.439426 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:46.439432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:46.439493 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:46.439579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:46.441327 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.442218 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.444054 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:46.444618 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:46.444735 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:46.448116 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:46.448132 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:46.448150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:46.448197 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:46.451189 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.452231 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.452241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.452244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.452470 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:46.452903 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:46.452907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:46.452909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:46.453313 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:46.453534 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:46.453991 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:40:46.454179 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:40:46.455340 15300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:46.511298 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.510806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40728)
0926 03:40:46.511024 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.511114 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 03:40:46.512293 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.511916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40732)
0926 03:40:46.512106 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.512185 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:40:46.580312 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15174
I20260926 03:40:46.597985 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 14796
I20260926 03:40:46.599205 14859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:46.712960 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:46.712391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33823 (local address 127.0.0.1:40740)
0926 03:40:46.712638 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:46.712755 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:40:46.713255 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 14796
I20260926 03:40:46.731454 1910 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:40:47.077518 1910 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:40:47.077755 1910 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:40:47.415329 1910 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:40:47.415 UTC [13848] LOG: received smart shutdown request
2026-09-26 03:40:47.418 UTC [13848] LOG: background worker "logical replication launcher" (PID 13857) exited with exit code 1
2026-09-26 03:40:47.419 UTC [13852] LOG: shutting down
2026-09-26 03:40:47.627 UTC [13848] LOG: database system is shut down
2026-09-26T03:40:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53192 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:40:47 dist-test-slave-lqrk krb5kdc[15371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:40:47 dist-test-slave-lqrk krb5kdc[15371](info): set up 2 sockets
Sep 26 03:40:47 dist-test-slave-lqrk krb5kdc[15371](info): commencing operation
krb5kdc: starting...
W20260926 03:40:48.792080 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:40:48 dist-test-slave-lqrk krb5kdc[15371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:40:49.064472 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42883
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:49.411194 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:49.411576 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:49.411692 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:49.419346 15384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:49.419514 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:49.419579 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:49.419665 15384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:49.419730 15384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:49.430150 15384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:49.431664 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:49.432997 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:49.446550 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:49.446602 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:49.446635 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:49.447522 15384 server_base.cc:1034] running on GCE node
I20260926 03:40:49.448407 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:49.450006 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:49.451272 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790394049451201 us; error 58 us; skew 500 ppm
I20260926 03:40:49.451793 15384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:49 dist-test-slave-lqrk krb5kdc[15371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:49.461447 15384 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:40:49.463495 15384 webserver.cc:460] Webserver started at http://127.1.221.190:40885/ using document root <none> and password file <none>
I20260926 03:40:49.464288 15384 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:49.464434 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:49.464754 15384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:49.469054 15384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "70cc05ea9aadc9b49fcb81632aebde97"
server_key_iv: "4770240417edefe675b382f53f46a6e6"
server_key_version: "encryptionkey@0"
I20260926 03:40:49.470494 15384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "70cc05ea9aadc9b49fcb81632aebde97"
server_key_iv: "4770240417edefe675b382f53f46a6e6"
server_key_version: "encryptionkey@0"
I20260926 03:40:49.477609 15384 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 03:40:49.482340 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:49.484596 15384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:40:49.484838 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "70cc05ea9aadc9b49fcb81632aebde97"
server_key_iv: "4770240417edefe675b382f53f46a6e6"
server_key_version: "encryptionkey@0"
I20260926 03:40:49.485075 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:49.510195 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:49.515200 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:49.515558 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:49.536823 15384 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42883
I20260926 03:40:49.536929 15450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42883 every 8 connection(s)
I20260926 03:40:49.538288 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:40:49.543565 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15384
I20260926 03:40:49.543860 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:40:49.543916 15451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:49.544801 1910 external_mini_cluster.cc:1428] Setting key 5ae62fc0b087e39eb5e1ab4900c1f4bd
I20260926 03:40:49.555930 15451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
Sep 26 03:40:49 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:49.586117 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.562826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.1.221.190:42883)
0926 03:40:49.563751 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:49.563764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:49.563849 (+ 85us) server_negotiation.cc:407] Connection header received
0926 03:40:49.565617 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:49.565633 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:49.565981 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:49.566217 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:49.567885 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:49.569311 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:49.570918 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:49.571664 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:49.571873 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:49.576499 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:49.576510 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:49.576529 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:49.576579 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:49.580283 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:49.581287 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:49.581297 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:49.581299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:49.581503 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:49.581976 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:49.581980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:49.581982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:49.582742 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:49.582982 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:49.583672 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 03:40:49.584197 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":644,"thread_start_us":425,"threads_started":1}
I20260926 03:40:49.586143 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.561082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42883 (local address 127.0.0.1:44110)
0926 03:40:49.562247 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.562306 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:49.565060 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:49.566281 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:49.566289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:49.567159 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:49.567662 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:49.567688 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:49.569499 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:49.569504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:49.570757 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:49.570773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:49.571945 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:49.571952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:49.572151 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:49.573029 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:49.573060 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:49.576196 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:49.580492 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:49.580507 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:49.580526 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:49.581103 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:49.581640 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:49.581646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:49.581650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:49.581842 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:49.582909 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:49.582918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:49.583181 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 03:40:49.583513 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 03:40:49.584362 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":182,"thread_start_us":440,"threads_started":1}
I20260926 03:40:49.590799 15451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:40:49.592491 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.591895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44664)
0926 03:40:49.592158 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.592319 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 03:40:49.592909 15400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
I20260926 03:40:49.594980 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.594451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44670)
0926 03:40:49.594646 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.594759 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 03:40:49.595237 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:34167: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260926 03:40:49.603271 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:34167
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:40:49.644698 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } attempt: 1
I20260926 03:40:49.646288 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.645642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44672)
0926 03:40:49.645926 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.646098 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 03:40:49.647598 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.647120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44686)
0926 03:40:49.647289 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.647421 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:40:49.647889 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:34167: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
I20260926 03:40:49.716217 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } attempt: 2
I20260926 03:40:49.717900 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.717209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44698)
0926 03:40:49.717565 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.717710 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 03:40:49.719332 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.718882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44714)
0926 03:40:49.719047 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.719169 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:40:49.719671 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:34167: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
I20260926 03:40:49.811024 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } attempt: 3
I20260926 03:40:49.813175 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.812150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44722)
0926 03:40:49.812598 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.812960 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260926 03:40:49.815052 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.814490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44732)
0926 03:40:49.814716 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.814839 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 03:40:49.815433 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:34167: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
W20260926 03:40:49.951119 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:49.951510 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:49.951665 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:40:49.958724 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } attempt: 4
W20260926 03:40:49.959327 15458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:49.959486 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:49.959600 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:49.959683 15458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:49.959748 15458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:49.960391 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.959647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44742)
0926 03:40:49.960094 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.960226 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
I20260926 03:40:49.961850 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:49.961341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44744)
0926 03:40:49.961498 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:49.961613 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:40:49.962114 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:34167: Network error: Client connection negotiation failed: client connection to 127.1.221.189:34167: connect: Connection refused (error 111)
I20260926 03:40:49.970743 15458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:49.972116 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:49.973413 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:49.986907 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:49.987051 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:49.986948 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:49.987722 15458 server_base.cc:1034] running on GCE node
I20260926 03:40:49.988806 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:49.990865 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:49.992216 15458 hybrid_clock.cc:648] HybridClock initialized: now 1790394049992139 us; error 58 us; skew 500 ppm
I20260926 03:40:49.992715 15458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:49 dist-test-slave-lqrk krb5kdc[15371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:49.996769 15458 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260926 03:40:49.998826 15458 webserver.cc:460] Webserver started at http://127.1.221.189:43223/ using document root <none> and password file <none>
I20260926 03:40:49.999845 15458 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:49.999975 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:50.000353 15458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:50.004505 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data/instance:
uuid: "84f06df9436440c18738f6ccd8af7ffe"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "ca1f3ebe7bd94acc546995fe2f9e219e"
server_key_iv: "7ca43c8ff8a9a5c77c70fb4733316dbf"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.005617 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal/instance:
uuid: "84f06df9436440c18738f6ccd8af7ffe"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "ca1f3ebe7bd94acc546995fe2f9e219e"
server_key_iv: "7ca43c8ff8a9a5c77c70fb4733316dbf"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.013005 15458 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 03:40:50.018167 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:50.020221 15458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:40:50.020432 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "84f06df9436440c18738f6ccd8af7ffe"
format_stamp: "Formatted at 2026-09-26 03:40:49 on dist-test-slave-lqrk"
server_key: "ca1f3ebe7bd94acc546995fe2f9e219e"
server_key_iv: "7ca43c8ff8a9a5c77c70fb4733316dbf"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.020663 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:50.047855 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:50.052490 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:50.052768 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:50.075685 15458 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:34167
I20260926 03:40:50.075755 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:34167 every 8 connection(s)
I20260926 03:40:50.077277 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
I20260926 03:40:50.082471 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15458
I20260926 03:40:50.082665 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-1/wal/instance
I20260926 03:40:50.082978 15525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:50.083176 1910 external_mini_cluster.cc:1428] Setting key e035149451f360e67e43bfd405b40bb4
I20260926 03:40:50.093626 15525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:50.122520 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.098225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.1.221.190:42883)
0926 03:40:50.098954 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.098962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.100208 (+ 1246us) server_negotiation.cc:407] Connection header received
0926 03:40:50.102450 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.102457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.102589 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.102783 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.105085 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.106450 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.109029 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.109634 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.109811 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.115141 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.115148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.115152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.115218 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.118422 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.119591 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.119595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.119598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.119762 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.120256 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.120260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.120262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.120615 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.120788 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.121753 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.122054 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":359,"threads_started":1}
I20260926 03:40:50.123965 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.099006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42883 (local address 127.0.0.1:44116)
0926 03:40:50.099990 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.100054 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.102092 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.103124 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.103134 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.103905 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.104767 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.104797 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.106945 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.106951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.108757 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.108782 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.110015 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.110022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.110282 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.111283 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.111317 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.114720 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.118700 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.118712 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.118734 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.119422 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.119892 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.119897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.119901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.120127 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.120765 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.120775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.121289 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.121617 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.122361 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":424,"threads_started":1}
I20260926 03:40:50.126528 15525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260926 03:40:50.143800 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:46223
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:40:50.145877 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.127558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44752)
0926 03:40:50.127819 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.127843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.127971 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.129377 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.129382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.129909 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.130067 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.130077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.131768 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.131774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.132725 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.132739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.134002 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.134009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.134330 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.135056 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.135077 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.137954 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.142437 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.142444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.142448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.143001 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.143684 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.143689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.143691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.143889 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.144742 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.144749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.144969 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.145142 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.145475 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:40:50.146066 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.127945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.1.221.189:34167)
0926 03:40:50.128813 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.128821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.128854 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:40:50.128920 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.128927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.129197 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.129377 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.130238 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.131612 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.132884 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.133780 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.133952 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.138299 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.138321 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.138342 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.138383 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.142166 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.143226 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.143239 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.143243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.143512 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.144021 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.144026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.144029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.144546 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.144788 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.145353 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.145646 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":469,"threads_started":1}
I20260926 03:40:50.149165 15525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:40:50.150671 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.149995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56822)
0926 03:40:50.150254 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.150453 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 03:40:50.151048 15477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.153057 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.152596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56830)
0926 03:40:50.152785 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.152905 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:40:50.153347 15525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.202797 15525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 1
I20260926 03:40:50.204550 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.203824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56846)
0926 03:40:50.204144 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.204330 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 03:40:50.206007 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.205544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56848)
0926 03:40:50.205754 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.205864 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 03:40:50.206338 15525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.261472 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } attempt: 5
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:40:50.274639 15525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 2
I20260926 03:40:50.276335 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.275644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56856)
0926 03:40:50.275946 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.276076 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 03:40:50.277907 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.277354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56858)
0926 03:40:50.277564 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.277692 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:40:50.278295 15525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.278476 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.262664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.1.221.189:34167)
0926 03:40:50.262887 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.262893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.262920 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:40:50.263137 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.263143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.263261 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.263354 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.264349 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.265133 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.266462 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.266967 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.267099 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.271204 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.271210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.271214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.271266 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.274554 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.275926 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.275932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.275935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.276124 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.276658 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.276664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.276667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.277071 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.277238 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.277837 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.278099 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:40:50.278455 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.262466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44762)
0926 03:40:50.262815 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.262841 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.262998 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.263440 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.263445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.263945 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.264170 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.264182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.265282 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.265288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.266304 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.266319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.267152 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.267158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.267439 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.268270 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.268292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.270923 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.274876 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.274882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.274886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.275679 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.276300 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.276306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.276310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.276513 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.277308 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.277313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.277514 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.277684 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.278057 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:40:50.279985 15451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:40:50.281232 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.280763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56872)
0926 03:40:50.280983 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.281072 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:40:50.282155 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.281799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56888)
0926 03:40:50.281943 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.282032 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:40:50.282356 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.333608 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 1
I20260926 03:40:50.334899 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.334404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56892)
0926 03:40:50.334606 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.334712 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:40:50.335871 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.335556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56902)
0926 03:40:50.335694 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.335762 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:40:50.336059 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.369577 15525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 3
I20260926 03:40:50.370870 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.370316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56906)
0926 03:40:50.370561 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.370674 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:40:50.372112 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.371618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56918)
0926 03:40:50.371859 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.371957 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260926 03:40:50.372331 15525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.404282 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 2
I20260926 03:40:50.405440 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.404952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56926)
0926 03:40:50.405188 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.405274 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:40:50.406730 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.406283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56940)
0926 03:40:50.406513 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.406600 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 03:40:50.406978 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
W20260926 03:40:50.489490 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:50.489815 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:50.489928 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:50.497880 15532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:50.498016 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:50.498066 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:50.498160 15532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:50.498224 15532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:50.508697 15532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42883,127.1.221.189:34167,127.1.221.188:46223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:50.510092 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:50.511480 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:40:50.513217 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 3
I20260926 03:40:50.514487 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.513972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56942)
0926 03:40:50.514202 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.514298 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 03:40:50.515534 15525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 4
I20260926 03:40:50.515717 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.515241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56954)
0926 03:40:50.515500 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.515589 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260926 03:40:50.515983 15451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
I20260926 03:40:50.516651 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.516138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56960)
0926 03:40:50.516352 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.516445 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:40:50.517863 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.517460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56972)
0926 03:40:50.517642 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.517736 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:40:50.518129 15525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46223: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46223: connect: Connection refused (error 111)
W20260926 03:40:50.524581 15539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:50.524545 15538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:50.524616 15541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:50.525470 15532 server_base.cc:1034] running on GCE node
I20260926 03:40:50.526396 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:50.527977 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:50.529188 15532 hybrid_clock.cc:648] HybridClock initialized: now 1790394050529147 us; error 37 us; skew 500 ppm
I20260926 03:40:50.529546 15532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394050, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:50.536379 15532 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260926 03:40:50.541326 15532 webserver.cc:460] Webserver started at http://127.1.221.188:42863/ using document root <none> and password file <none>
I20260926 03:40:50.542649 15532 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:50.542829 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:50.543192 15532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:50.548250 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data/instance:
uuid: "03dcc917511c403f8b41840431c75215"
format_stamp: "Formatted at 2026-09-26 03:40:50 on dist-test-slave-lqrk"
server_key: "ec302ccb1eed68dcbce54778b38d5462"
server_key_iv: "9bb5f04fdc731c1ee75b91f97f2600c9"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.549731 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal/instance:
uuid: "03dcc917511c403f8b41840431c75215"
format_stamp: "Formatted at 2026-09-26 03:40:50 on dist-test-slave-lqrk"
server_key: "ec302ccb1eed68dcbce54778b38d5462"
server_key_iv: "9bb5f04fdc731c1ee75b91f97f2600c9"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.557062 15532 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:40:50.562457 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:50.564462 15532 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:40:50.564709 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "03dcc917511c403f8b41840431c75215"
format_stamp: "Formatted at 2026-09-26 03:40:50 on dist-test-slave-lqrk"
server_key: "ec302ccb1eed68dcbce54778b38d5462"
server_key_iv: "9bb5f04fdc731c1ee75b91f97f2600c9"
server_key_version: "encryptionkey@0"
I20260926 03:40:50.565007 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:50.597023 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:50.601414 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:50.601691 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:50.619853 15532 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:46223
I20260926 03:40:50.619910 15599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:46223 every 8 connection(s)
I20260926 03:40:50.621392 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
I20260926 03:40:50.625423 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:50.630326 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15532
I20260926 03:40:50.630535 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393848592298-1910-0/minicluster-data/master-2/wal/instance
I20260926 03:40:50.630975 1910 external_mini_cluster.cc:1428] Setting key c61a06e134c742f696cf6d5299a77e48
I20260926 03:40:50.632580 15600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394050, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:40:50.654085 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.635276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.1.221.190:42883)
0926 03:40:50.635756 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.635763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.636306 (+ 543us) server_negotiation.cc:407] Connection header received
0926 03:40:50.636597 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.636603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.636701 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.636812 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.638470 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.639687 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.641867 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.642496 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.642645 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.647161 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.647167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.647171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.647219 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.650482 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.651435 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.651441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.651445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.651603 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.652100 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.652104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.652106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.652594 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.652703 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.653458 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.653670 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260926 03:40:50.655197 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.632900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.1.221.188:46223)
0926 03:40:50.633730 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.633742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.633786 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:40:50.635904 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.635928 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.636361 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.636639 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.638442 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.640173 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.642028 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.642928 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.643041 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.647838 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.647848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.647864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.647907 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.651320 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.652238 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.652247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.652251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.652400 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.652753 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.652756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.652758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.653142 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.653370 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.653773 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.654024 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"spinlock_wait_cycles":25344,"thread_start_us":393,"threads_started":1}
I20260926 03:40:50.655237 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.635297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42883 (local address 127.0.0.1:44128)
0926 03:40:50.636145 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.636194 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.636361 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.636899 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.636906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.637721 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.638279 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.638302 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.639858 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.639862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.641682 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.641699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.642749 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.642758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.642993 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.643878 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.643972 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.646885 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.650666 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.650676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.650687 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.651271 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.651716 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.651720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.651723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.651969 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.652727 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.652736 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.653096 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.653350 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.653884 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":456,"threads_started":1}
I20260926 03:40:50.657068 15600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394050, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:50.671046 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.657899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:34167 (local address 127.0.0.1:44772)
0926 03:40:50.658206 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.658221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.658343 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.658715 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.658720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.659114 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.659268 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.659277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.660486 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.660492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.661404 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.661417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.662129 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.662135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.662337 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.662916 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.662933 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.665231 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.668611 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.668616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.668619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.669033 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.669409 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.669414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.669418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.669576 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.670067 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.670072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.670234 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.670354 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.670675 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260926 03:40:50.671108 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.657912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.1.221.189:34167)
0926 03:40:50.658099 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.658104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.658296 (+ 192us) server_negotiation.cc:407] Connection header received
0926 03:40:50.658474 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.658480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.658568 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.658672 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.659449 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.660316 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.661545 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.661970 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.662091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.665498 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.665506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.665510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.665551 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.668432 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.669156 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.669161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.669163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.669300 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.669679 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.669683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.669685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.669957 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.670122 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.670456 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.670692 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:40:50.672549 15600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:40:50.674015 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.658757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.1.221.190:42883)
0926 03:40:50.658924 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.658929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.658947 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:40:50.659112 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.659118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.659191 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.659272 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.660550 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.661240 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.662521 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.663079 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.663193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.667760 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.667765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.667769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.667806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.670350 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.671108 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.671113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.671117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.671270 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.671661 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.671666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.671669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.672072 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.673443 (+ 1371us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.673576 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.673772 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394050, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:40:50.686450 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.673276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:56994)
0926 03:40:50.673459 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.673478 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.673592 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.673837 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.673841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.674289 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.674444 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.674453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.675472 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.675477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.676773 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.676801 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.677504 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.677510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.677696 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.678354 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.678373 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.680958 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.683839 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.683843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.683847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.684407 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.684788 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.684793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.684797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.684928 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.685467 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.685471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.685616 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.685743 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.686042 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:40:50.686503 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.673320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.1.221.188:46223)
0926 03:40:50.673459 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.673465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.673521 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:40:50.673683 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.673687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.673737 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.673810 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.674563 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.675282 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.676932 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.677351 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.677441 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.681144 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.681148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.681152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.681182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.683691 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.684532 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.684538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.684540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.684674 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.685034 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.685038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.685040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.685356 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.685498 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.685850 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.686218 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":39}
I20260926 03:40:50.691116 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.675659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.1.221.189:34167)
0926 03:40:50.675813 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.675817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.675914 (+ 97us) server_negotiation.cc:407] Connection header received
0926 03:40:50.676088 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.676091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.676148 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.676218 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.677646 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.678477 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.680182 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.680740 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.680852 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.685205 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.685210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.685214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.685251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.688098 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.689073 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.689079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.689082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.689238 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.689785 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.689789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.689791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.690079 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.690282 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.690628 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.690821 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:40:50.692502 15600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: Bootstrap starting.
I20260926 03:40:50.693226 15451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 4
I20260926 03:40:50.696496 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:40:50.696856 15600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:50.697957 15600 log.cc:824] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:40:50.701812 15600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: No bootstrap required, opened a new log
I20260926 03:40:50.708158 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.693844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:47910)
0926 03:40:50.694102 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.694123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.694242 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.694599 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.694605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.695048 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.695271 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.695285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.696730 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.696735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.698082 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.698100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.699133 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.699139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.699328 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.699960 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.699979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.702234 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.705520 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.705526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.705530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.705933 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.706338 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.706343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.706347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.706490 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.707097 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.707102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.707254 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.707419 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.707756 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 03:40:50.708175 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.693996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.1.221.188:46223)
0926 03:40:50.694212 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.694221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.694241 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:40:50.694371 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.694375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.694460 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.694551 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.695469 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.696503 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.698240 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.698953 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.699081 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.702446 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.702452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.702455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.702491 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.705337 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.706061 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.706066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.706070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.706226 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.706602 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.706607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.706610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.706958 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.707128 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.707542 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.707761 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:40:50.713376 15451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: Bootstrap starting.
I20260926 03:40:50.714659 15600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.715193 15600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:50.715308 15600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03dcc917511c403f8b41840431c75215, State: Initialized, Role: FOLLOWER
I20260926 03:40:50.716181 15600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.716903 15451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:50.717283 15611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:50.717784 15611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:50.718681 15600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:50.719506 15451 log.cc:824] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:50.731356 15451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: No bootstrap required, opened a new log
I20260926 03:40:50.744748 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.718362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.1.221.189:34167)
0926 03:40:50.718539 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.718545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.726130 (+ 7585us) server_negotiation.cc:407] Connection header received
0926 03:40:50.726467 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.726473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.726560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.726633 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.729637 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.730681 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.732791 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.733434 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.733544 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.737586 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.737593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.737598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.737648 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.741436 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.742332 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.742338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.742342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.742498 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.742917 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.742922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.742926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.743310 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.743562 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.744136 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.744386 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:40:50.747118 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.725373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.1.221.188:46223)
0926 03:40:50.725664 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.725671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.729007 (+ 3336us) server_negotiation.cc:407] Connection header received
0926 03:40:50.729155 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.729162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.729281 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.729390 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.731699 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.732848 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.734999 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.735803 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.735909 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.739681 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.739689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.739694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.739746 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.743551 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.744517 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.744524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.744529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.744709 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.745172 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.745178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.745181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.745604 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.745881 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.746447 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.746732 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:40:50.750692 15451 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.751175 15451 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:50.751295 15451 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bbcf29e8b742d8a6323dd67c9ef0b7, State: Initialized, Role: FOLLOWER
I20260926 03:40:50.752084 15451 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.753496 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:50.753964 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:50.754971 15451 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:50.755827 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.717400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.1.221.190:42883)
0926 03:40:50.717599 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.717605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.726364 (+ 8759us) server_negotiation.cc:407] Connection header received
0926 03:40:50.726539 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.726544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.726634 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.726723 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.731522 (+ 4799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.732625 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.737501 (+ 4876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.738153 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.738259 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.747589 (+ 9330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.747596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.747601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.747647 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.751219 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.752218 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.752225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.752229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.752452 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.752956 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.752961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.752965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.753487 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.754972 (+ 1485us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.755130 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.755421 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 03:40:50.769233 15405 tablet.cc:2369] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:50.766883 15384 master_runner.cc:186] Error getting master registration for 127.1.221.189:34167: OK, instance_id { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" instance_seqno: 1790394050069379 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:40:50.779829 15384 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 03:40:50.781180 15638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:40:50.781329 15638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 03:40:50.784305 15635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:40:50.784562 15635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:40:50.772324 15532 master_runner.cc:186] Error getting master registration for 127.1.221.189:34167: OK, instance_id { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" instance_seqno: 1790394050069379 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:40:50.793727 15532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:40:50.817379 15525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } attempt: 5
Sep 26 03:40:50 dist-test-slave-lqrk krb5kdc[15371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394049, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:40:50.837102 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.818219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46223 (local address 127.0.0.1:47926)
0926 03:40:50.818604 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:50.818627 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:50.818803 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:50.819208 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:50.819213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:50.819816 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:50.820042 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.820056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.821636 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.821643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.823151 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:50.823171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.825454 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.825467 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:50.825722 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.826453 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:50.826480 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:50.829429 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:50.833938 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.833945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.833949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.834569 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.835054 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:50.835060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:50.835064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:50.835243 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:50.835938 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:50.835943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:50.836124 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 03:40:50.836284 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:40:50.836672 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260926 03:40:50.837131 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:50.818362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.1.221.188:46223)
0926 03:40:50.818606 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:50.818613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:50.818709 (+ 96us) server_negotiation.cc:407] Connection header received
0926 03:40:50.818951 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:50.818958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:50.819049 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:50.819169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:50.820254 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.821441 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.823426 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:50.824181 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:50.824316 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:50.829729 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:50.829738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:50.829743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:50.829790 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:50.833710 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.834730 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.834737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.834741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.834930 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:50.835367 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:50.835373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:50.835404 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:50.835799 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:50.835965 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:50.836422 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:40:50.836715 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:40:50.844671 15525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: Bootstrap starting.
I20260926 03:40:50.849041 15525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:50.850481 15525 log.cc:824] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: Log is configured to *not* fsync() on all Append() calls
W20260926 03:40:50.854061 15554 tablet.cc:2369] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:50.854167 15525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: No bootstrap required, opened a new log
I20260926 03:40:50.867581 15525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.868086 15525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:50.868264 15525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f06df9436440c18738f6ccd8af7ffe, State: Initialized, Role: FOLLOWER
I20260926 03:40:50.869061 15525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:50.872473 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:50.873037 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:50.874195 15525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:50.893761 15458 master_runner.cc:263] No leader master found from master 84f06df9436440c18738f6ccd8af7ffe
I20260926 03:40:50.893939 15458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 03:40:50.898460 15653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:40:50.898764 15653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:40:51.031690 15611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:40:51.032105 15611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:51.034143 15611 leader_election.cc:288] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3bbcf29e8b742d8a6323dd67c9ef0b7 (127.1.221.190:42883), 84f06df9436440c18738f6ccd8af7ffe (127.1.221.189:34167)
I20260926 03:40:51.035661 15500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dcc917511c403f8b41840431c75215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f06df9436440c18738f6ccd8af7ffe" is_pre_election: true
I20260926 03:40:51.036489 15500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03dcc917511c403f8b41840431c75215 in term 0.
I20260926 03:40:51.037535 15549 leader_election.cc:302] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03dcc917511c403f8b41840431c75215, 84f06df9436440c18738f6ccd8af7ffe; no voters:
I20260926 03:40:51.038239 15611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:40:51.038512 15611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:40:51.038643 15611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:51.039492 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dcc917511c403f8b41840431c75215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" is_pre_election: true
I20260926 03:40:51.040174 15426 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03dcc917511c403f8b41840431c75215 in term 0.
I20260926 03:40:51.042255 15611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:51.044142 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dcc917511c403f8b41840431c75215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7"
I20260926 03:40:51.044412 15426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:51.048462 15426 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03dcc917511c403f8b41840431c75215 in term 1.
I20260926 03:40:51.049373 15552 leader_election.cc:302] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03dcc917511c403f8b41840431c75215, d3bbcf29e8b742d8a6323dd67c9ef0b7; no voters:
I20260926 03:40:51.054180 15500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dcc917511c403f8b41840431c75215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f06df9436440c18738f6ccd8af7ffe"
I20260926 03:40:51.054455 15500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:40:51.056147 15479 tablet.cc:2369] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:51.059732 15500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03dcc917511c403f8b41840431c75215 in term 1.
I20260926 03:40:51.060571 15611 leader_election.cc:288] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [CANDIDATE]: Term 1 election: Requested vote from peers d3bbcf29e8b742d8a6323dd67c9ef0b7 (127.1.221.190:42883), 84f06df9436440c18738f6ccd8af7ffe (127.1.221.189:34167)
I20260926 03:40:51.060928 15611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:51.061233 15611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [term 1 LEADER]: Becoming Leader. State: Replica: 03dcc917511c403f8b41840431c75215, State: Running, Role: LEADER
I20260926 03:40:51.062057 15611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } }
I20260926 03:40:51.071915 15611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dcc917511c403f8b41840431c75215. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.075006 15611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:51.082279 15657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:40:51.084607 15657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:40:51.105181 15500 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [term 1 FOLLOWER]: Refusing update from remote peer 03dcc917511c403f8b41840431c75215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:51.106572 15611 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:51.115747 15426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [term 1 FOLLOWER]: Refusing update from remote peer 03dcc917511c403f8b41840431c75215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:51.117229 15611 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:51.151889 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dcc917511c403f8b41840431c75215. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.152240 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:51.156018 15658 mvcc.cc:205] Tried to move back new op lower bound from 7333454033315500032 to 7333454033164877824. Current Snapshot: MvccSnapshot[applied={T|T < 7333454033315500032}]
I20260926 03:40:51.155870 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dcc917511c403f8b41840431c75215. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.156180 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:51.157933 15659 mvcc.cc:205] Tried to move back new op lower bound from 7333454033315500032 to 7333454033164877824. Current Snapshot: MvccSnapshot[applied={T|T < 7333454033315500032}]
I20260926 03:40:51.168046 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.168371 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:51.190251 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.190570 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:40:51.194169 15655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.194540 15655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:51.194921 15655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03dcc917511c403f8b41840431c75215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bbcf29e8b742d8a6323dd67c9ef0b7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42883 } } peers { permanent_uuid: "84f06df9436440c18738f6ccd8af7ffe" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 34167 } } peers { permanent_uuid: "03dcc917511c403f8b41840431c75215" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46223 } } }
I20260926 03:40:51.195127 15655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:51.195571 15657 catalog_manager.cc:1347] Generated new cluster ID: 67182e37892f46918e7d62b88b942bbf
I20260926 03:40:51.195686 15657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:40:51.223308 15657 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:40:51.225291 15657 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:40:51.253214 15657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03dcc917511c403f8b41840431c75215: Generated new TSK 0
I20260926 03:40:51.254406 15657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:40:51.786712 15635 catalog_manager.cc:1259] Loaded cluster ID: 67182e37892f46918e7d62b88b942bbf
I20260926 03:40:51.786825 15635 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: loading cluster ID for follower catalog manager: success
I20260926 03:40:51.789691 15635 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: acquiring CA information for follower catalog manager: success
I20260926 03:40:51.791831 15635 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3bbcf29e8b742d8a6323dd67c9ef0b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:40:51.900872 15653 catalog_manager.cc:1259] Loaded cluster ID: 67182e37892f46918e7d62b88b942bbf
I20260926 03:40:51.900993 15653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: loading cluster ID for follower catalog manager: success
I20260926 03:40:51.903483 15653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: acquiring CA information for follower catalog manager: success
I20260926 03:40:51.905804 15653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84f06df9436440c18738f6ccd8af7ffe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:40:52.372007 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15384
I20260926 03:40:52.373600 15446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:52.490069 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15384
I20260926 03:40:52.513128 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15458
I20260926 03:40:52.514642 15520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:52.627111 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15458
W20260926 03:40:52.645654 15549 connection.cc:535] client connection to 127.1.221.189:34167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 03:40:52.645973 15549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:40:52.646023 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15532
I20260926 03:40:52.647516 15595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:52.775208 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15532
2026-09-26T03:40:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5034 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:40:52 dist-test-slave-lqrk krb5kdc[15677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:40:52 dist-test-slave-lqrk krb5kdc[15677](info): set up 2 sockets
Sep 26 03:40:52 dist-test-slave-lqrk krb5kdc[15677](info): commencing operation
krb5kdc: starting...
W20260926 03:40:53.824611 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:40:53 dist-test-slave-lqrk krb5kdc[15677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:40:54.103825 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41001
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:54.511744 15690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:54.512101 15690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:54.512226 15690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:54.520052 15690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:54.520215 15690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:54.520272 15690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:54.520364 15690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:54.520431 15690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:54.531178 15690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:54.532730 15690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:54.534229 15690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:54.549223 15695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:54.549381 15696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:54.549360 15698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:54.549981 15690 server_base.cc:1034] running on GCE node
I20260926 03:40:54.551222 15690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:54.553162 15690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:54.554459 15690 hybrid_clock.cc:648] HybridClock initialized: now 1790394054554395 us; error 55 us; skew 500 ppm
I20260926 03:40:54.554946 15690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:54 dist-test-slave-lqrk krb5kdc[15677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:54.562183 15690 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:40:54.564289 15690 webserver.cc:460] Webserver started at http://127.1.221.190:36239/ using document root <none> and password file <none>
I20260926 03:40:54.565126 15690 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:54.565264 15690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:54.565614 15690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:54.569059 15690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d"
format_stamp: "Formatted at 2026-09-26 03:40:54 on dist-test-slave-lqrk"
server_key: "23feda510e6e74b4ae4b5c5629d77664"
server_key_iv: "6a45a11943f482495ea5b7aa517233eb"
server_key_version: "encryptionkey@0"
I20260926 03:40:54.569900 15690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d"
format_stamp: "Formatted at 2026-09-26 03:40:54 on dist-test-slave-lqrk"
server_key: "23feda510e6e74b4ae4b5c5629d77664"
server_key_iv: "6a45a11943f482495ea5b7aa517233eb"
server_key_version: "encryptionkey@0"
I20260926 03:40:54.576671 15690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 03:40:54.581703 15705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:54.583832 15690 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:40:54.584087 15690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d"
format_stamp: "Formatted at 2026-09-26 03:40:54 on dist-test-slave-lqrk"
server_key: "23feda510e6e74b4ae4b5c5629d77664"
server_key_iv: "6a45a11943f482495ea5b7aa517233eb"
server_key_version: "encryptionkey@0"
I20260926 03:40:54.584323 15690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:54.612681 15690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:54.617408 15690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:54.617704 15690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:54.638753 15690 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41001
I20260926 03:40:54.638819 15756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41001 every 8 connection(s)
I20260926 03:40:54.640430 15690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:40:54.643316 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15690
I20260926 03:40:54.643560 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:40:54.644233 1910 external_mini_cluster.cc:1428] Setting key 09d4f07b24445e9e8461767c03fd5c4e
I20260926 03:40:54.646420 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:40:54 dist-test-slave-lqrk krb5kdc[15677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:54.661090 15757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d: Bootstrap starting.
I20260926 03:40:54.665789 15757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:54.667514 15757 log.cc:824] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:54.672052 15757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d: No bootstrap required, opened a new log
I20260926 03:40:54.672096 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:54.646463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.1.221.190:41001)
0926 03:40:54.647591 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:54.647627 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:54.647689 (+ 62us) server_negotiation.cc:407] Connection header received
0926 03:40:54.649574 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:54.649607 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:54.650160 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:54.650651 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:54.652573 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:54.654122 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:54.655736 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:54.656594 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:54.656819 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:54.662105 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:54.662117 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:54.662143 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:54.662199 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:54.665617 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:54.667010 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:54.667020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:54.667030 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:54.667464 (+ 434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:54.668046 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:54.668050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:54.668052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:54.669016 (+ 964us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:54.669379 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:54.670114 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 03:40:54.670692 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":444,"threads_started":1}
I20260926 03:40:54.677482 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:40:54.682204 15757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } }
I20260926 03:40:54.682760 15757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:54.682857 15757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81bc82e40aec4e1cbf5975f6c9fa7c8d, State: Initialized, Role: FOLLOWER
I20260926 03:40:54.683627 15757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } }
I20260926 03:40:54.683911 15757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:40:54.684052 15757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:40:54.684242 15757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:54.686931 15757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } }
I20260926 03:40:54.687743 15757 leader_election.cc:302] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81bc82e40aec4e1cbf5975f6c9fa7c8d; no voters:
I20260926 03:40:54.688637 15757 leader_election.cc:288] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:40:54.688885 15762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:54.689774 15762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [term 1 LEADER]: Becoming Leader. State: Replica: 81bc82e40aec4e1cbf5975f6c9fa7c8d, State: Running, Role: LEADER
I20260926 03:40:54.690558 15762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } }
I20260926 03:40:54.691419 15757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:54.694473 15764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81bc82e40aec4e1cbf5975f6c9fa7c8d. Latest consensus state: current_term: 1 leader_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } } }
I20260926 03:40:54.695150 15764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:54.694697 15763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bc82e40aec4e1cbf5975f6c9fa7c8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41001 } } }
I20260926 03:40:54.696321 15763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:54.700369 15771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:40:54.706215 15771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:40:54.718328 15771 catalog_manager.cc:1347] Generated new cluster ID: eb680b38a4c24acdb03d3d3d99782bf2
I20260926 03:40:54.718516 15771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:40:54.744985 15771 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:40:54.747210 15771 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:40:54.761916 15771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81bc82e40aec4e1cbf5975f6c9fa7c8d: Generated new TSK 0
I20260926 03:40:54.763144 15771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:40:54.794361 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15690
I20260926 03:40:54.796154 15752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:54.895753 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15690
2026-09-26T03:40:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2115 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:40:54 dist-test-slave-lqrk krb5kdc[15785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:40:54 dist-test-slave-lqrk krb5kdc[15785](info): set up 2 sockets
Sep 26 03:40:54 dist-test-slave-lqrk krb5kdc[15785](info): commencing operation
krb5kdc: starting...
W20260926 03:40:55.940033 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:40:56 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:40:56.220144 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:56.633519 15798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:56.633872 15798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:56.633965 15798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:56.642087 15798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:40:56.642268 15798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:56.642366 15798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:56.642484 15798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:40:56.642578 15798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:40:56.653770 15798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:56.655336 15798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:56.656883 15798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:56.671324 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:56.671427 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:56.671481 15803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:56.672757 15798 server_base.cc:1034] running on GCE node
I20260926 03:40:56.674175 15798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:56.676249 15798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:56.677668 15798 hybrid_clock.cc:648] HybridClock initialized: now 1790394056677557 us; error 68 us; skew 500 ppm
I20260926 03:40:56.678164 15798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:56 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:56.682204 15798 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:40:56.684502 15798 webserver.cc:460] Webserver started at http://127.1.221.190:36081/ using document root <none> and password file <none>
I20260926 03:40:56.685438 15798 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:56.685586 15798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:56.685931 15798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:56.689728 15798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "5162b4a1bb5248aaa9d5615d962715e6"
format_stamp: "Formatted at 2026-09-26 03:40:56 on dist-test-slave-lqrk"
server_key: "b3a9423ae6ebcfbc0abbaa93307cb1ee"
server_key_iv: "fee04a67e9699e645471cacbd201e67d"
server_key_version: "encryptionkey@0"
I20260926 03:40:56.690685 15798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "5162b4a1bb5248aaa9d5615d962715e6"
format_stamp: "Formatted at 2026-09-26 03:40:56 on dist-test-slave-lqrk"
server_key: "b3a9423ae6ebcfbc0abbaa93307cb1ee"
server_key_iv: "fee04a67e9699e645471cacbd201e67d"
server_key_version: "encryptionkey@0"
I20260926 03:40:56.698736 15798 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260926 03:40:56.704053 15813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:56.706350 15798 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:40:56.706610 15798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "5162b4a1bb5248aaa9d5615d962715e6"
format_stamp: "Formatted at 2026-09-26 03:40:56 on dist-test-slave-lqrk"
server_key: "b3a9423ae6ebcfbc0abbaa93307cb1ee"
server_key_iv: "fee04a67e9699e645471cacbd201e67d"
server_key_version: "encryptionkey@0"
I20260926 03:40:56.706902 15798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:56.728669 15798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:56.733543 15798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:56.733907 15798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:56.754865 15798 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34117
I20260926 03:40:56.754944 15864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34117 every 8 connection(s)
I20260926 03:40:56.756500 15798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:40:56.761955 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:56.763693 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15798
I20260926 03:40:56.763988 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:40:56.765120 1910 external_mini_cluster.cc:1428] Setting key 99836810ccc1e596209180b91a569bc4
I20260926 03:40:56.780663 15865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap starting.
Sep 26 03:40:56 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:56.786885 15865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:56.788501 15865 log.cc:824] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:56.792557 15865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: No bootstrap required, opened a new log
I20260926 03:40:56.794641 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:56.768040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.1.221.190:34117)
0926 03:40:56.769085 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:56.769108 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:56.769164 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:40:56.771032 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:56.771062 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:56.771662 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:56.772110 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:56.774324 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:56.775934 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:56.778092 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:56.778997 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:56.779216 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:56.786152 (+ 6936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:56.786164 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:56.786187 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:56.786254 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:56.789826 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:56.790721 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:56.790731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:56.790739 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:56.790945 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:56.791363 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:56.791368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:56.791370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:56.792050 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:56.792399 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:56.792866 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 03:40:56.793413 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":489,"threads_started":1}
I20260926 03:40:56.802168 15865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:40:56.802810 15865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:56.802968 15865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Initialized, Role: FOLLOWER
I20260926 03:40:56.803807 15865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:40:56.804174 15865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:40:56.804370 15865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:40:56.804637 15865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:56.807790 15865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:40:56.808506 15865 leader_election.cc:302] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5162b4a1bb5248aaa9d5615d962715e6; no voters:
I20260926 03:40:56.809342 15865 leader_election.cc:288] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:40:56.809604 15870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:56.810726 15870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 LEADER]: Becoming Leader. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Running, Role: LEADER
I20260926 03:40:56.811578 15870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:40:56.812186 15865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:40:56.815478 15872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5162b4a1bb5248aaa9d5615d962715e6. Latest consensus state: current_term: 1 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:40:56.815574 15871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:40:56.816116 15872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:56.816115 15871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:40:56.819659 15875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:40:56.826869 15875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:40:56.839148 15875 catalog_manager.cc:1347] Generated new cluster ID: 33460f8e96344dd6bea74971afd8857f
I20260926 03:40:56.839324 15875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:40:56.872031 15875 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:40:56.873900 15875 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:40:56.884747 15875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Generated new TSK 0
I20260926 03:40:56.886019 15875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:40:56.964841 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:57.399468 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:57.399904 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:57.400065 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:57.408635 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:57.408823 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:57.408995 15891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:40:57.421180 15891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:57.423148 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:57.425036 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:57.442708 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:57.442806 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:57.442746 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:57.449239 15891 server_base.cc:1034] running on GCE node
I20260926 03:40:57.450423 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:57.451949 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:57.453235 15891 hybrid_clock.cc:648] HybridClock initialized: now 1790394057453141 us; error 78 us; skew 500 ppm
I20260926 03:40:57.453689 15891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:57 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:57.458053 15891 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:40:57.460565 15891 webserver.cc:460] Webserver started at http://127.1.221.129:32929/ using document root <none> and password file <none>
I20260926 03:40:57.461570 15891 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:57.461720 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:57.462085 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:57.466001 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "277941c72bff47e9a2a26623a3cac6a1"
format_stamp: "Formatted at 2026-09-26 03:40:57 on dist-test-slave-lqrk"
server_key: "15b9182a15349f3772c1ec2dabc5de98"
server_key_iv: "e0a88f8eb4b139531d732e2a4c74f511"
server_key_version: "encryptionkey@0"
I20260926 03:40:57.467049 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "277941c72bff47e9a2a26623a3cac6a1"
format_stamp: "Formatted at 2026-09-26 03:40:57 on dist-test-slave-lqrk"
server_key: "15b9182a15349f3772c1ec2dabc5de98"
server_key_iv: "e0a88f8eb4b139531d732e2a4c74f511"
server_key_version: "encryptionkey@0"
I20260926 03:40:57.474766 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260926 03:40:57.479969 15906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:57.482121 15891 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:40:57.482379 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "277941c72bff47e9a2a26623a3cac6a1"
format_stamp: "Formatted at 2026-09-26 03:40:57 on dist-test-slave-lqrk"
server_key: "15b9182a15349f3772c1ec2dabc5de98"
server_key_iv: "e0a88f8eb4b139531d732e2a4c74f511"
server_key_version: "encryptionkey@0"
I20260926 03:40:57.482630 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:57.503679 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:57.508708 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:57.509006 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:57.510169 15891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:57.512260 15891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:57.512354 15891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:57.512513 15891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:57.512575 15891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:57.553628 15891 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43387
I20260926 03:40:57.553704 16018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43387 every 8 connection(s)
I20260926 03:40:57.555279 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:40:57.559291 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 15891
I20260926 03:40:57.559867 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:40:57.560755 1910 external_mini_cluster.cc:1428] Setting key 3f9332003f1eb51d58ebc60781eff4b2
Sep 26 03:40:57 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:57.583305 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:57.559225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44081 (local address 127.1.221.190:34117)
0926 03:40:57.560132 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:57.560140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:57.561317 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 03:40:57.563969 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:57.563975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:57.564065 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:57.564202 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:57.566484 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:57.567587 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:57.569732 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:57.570328 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:57.570494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:57.576032 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:57.576038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:57.576042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:57.576092 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:57.579018 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:57.580394 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:57.580399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:57.580401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:57.580569 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:57.581180 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:57.581183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:57.581185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:57.581487 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:57.581624 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:57.582776 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0926 03:40:57.583022 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":457,"threads_started":1}
I20260926 03:40:57.585240 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:57.559998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:44081)
0926 03:40:57.561056 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:57.561122 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:57.563527 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:57.564530 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:57.564542 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:57.565361 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:57.566215 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:57.566241 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:57.567804 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:57.567809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:57.569466 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:57.569489 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:57.570696 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:57.570703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:57.571088 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:57.572135 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:57.572188 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:57.575619 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:57.579319 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:57.579333 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:57.579359 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:57.580197 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:57.580724 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:57.580730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:57.580733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:57.581059 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:57.581616 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:57.581626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:57.582251 (+ 625us) client_negotiation.cc:769] Sending connection context
0926 03:40:57.582648 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 03:40:57.583420 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":382,"threads_started":1}
I20260926 03:40:57.587931 16019 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:40:57.588454 16019 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:57.589792 16019 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:40:57.593504 15830 ts_manager.cc:194] Registered new tserver with Master: 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387)
I20260926 03:40:57.595482 15830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:44081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:40:57.627172 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:40:58.043634 16025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:58.044035 16025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:58.044179 16025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:58.053051 16025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:58.053237 16025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:58.053411 16025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:40:58.064893 16025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:58.066641 16025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:58.068583 16025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:58.084618 16030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:58.084836 16033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:58.084419 16031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:58.086477 16025 server_base.cc:1034] running on GCE node
I20260926 03:40:58.087445 16025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:58.088879 16025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:58.090127 16025 hybrid_clock.cc:648] HybridClock initialized: now 1790394058090059 us; error 54 us; skew 500 ppm
I20260926 03:40:58.090585 16025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:58 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:58.094866 16025 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:40:58.096983 16025 webserver.cc:460] Webserver started at http://127.1.221.130:33281/ using document root <none> and password file <none>
I20260926 03:40:58.097908 16025 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:58.098032 16025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:58.098394 16025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:58.102150 16025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "9fd490db62ed4935a42f8ae3674c6ced"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "74a93cd8f4ede0d6a569c821061efa26"
server_key_iv: "a6480759008a6dbd1d880654f17ddace"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.103060 16025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "9fd490db62ed4935a42f8ae3674c6ced"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "74a93cd8f4ede0d6a569c821061efa26"
server_key_iv: "a6480759008a6dbd1d880654f17ddace"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.109531 16025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:40:58.114121 16040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:58.115973 16025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:40:58.116163 16025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "9fd490db62ed4935a42f8ae3674c6ced"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "74a93cd8f4ede0d6a569c821061efa26"
server_key_iv: "a6480759008a6dbd1d880654f17ddace"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.116379 16025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:58.138840 16025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:58.144052 16025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:58.144378 16025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:58.145692 16025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:58.147475 16025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:58.147567 16025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:40:58.147706 16025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:58.147763 16025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:58.184762 16025 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:46399
I20260926 03:40:58.184893 16152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:46399 every 8 connection(s)
I20260926 03:40:58.186355 16025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:40:58.188094 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16025
I20260926 03:40:58.188299 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:40:58.188912 1910 external_mini_cluster.cc:1428] Setting key 5e8316f2dec7cafc8f43e20b2c34d00c
Sep 26 03:40:58 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:58.212427 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.190056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41399 (local address 127.1.221.190:34117)
0926 03:40:58.190649 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.190656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.192024 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 03:40:58.194164 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.194171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.194263 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.194401 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:58.196246 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.197409 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.199230 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.199932 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.200074 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.204805 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:58.204813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:58.204818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:58.204870 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:58.207703 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.209099 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.209105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.209108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.209315 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.209964 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.209969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.209971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.210331 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:58.210469 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.211793 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.212123 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260926 03:40:58.214138 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.190798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:41399)
0926 03:40:58.191804 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:58.191867 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:58.193825 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:58.194650 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:58.194659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:58.195284 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:58.196030 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.196056 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.197618 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.197623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.199040 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.199056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.200196 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.200202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.200502 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.201237 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:58.201263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:58.204344 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:58.207968 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:58.207982 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:58.208010 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:58.208830 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:58.209455 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:58.209461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:58.209464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:58.209784 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:58.210480 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:58.210491 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:58.211134 (+ 643us) client_negotiation.cc:769] Sending connection context
0926 03:40:58.211529 (+ 395us) client_negotiation.cc:240] Negotiation successful
0926 03:40:58.212270 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":373,"threads_started":1}
I20260926 03:40:58.216693 16153 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:40:58.217267 16153 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:40:58.218286 16153 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260926 03:40:58.220606 15830 ts_manager.cc:194] Registered new tserver with Master: 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399)
I20260926 03:40:58.221730 15830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:41399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:40:58.259429 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:40:58.599869 16019 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
W20260926 03:40:58.655354 16159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:40:58.655709 16159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:40:58.655835 16159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:40:58.663358 16159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:40:58.663542 16159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:40:58.663690 16159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:40:58.673933 16159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:40:58.675467 16159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:40:58.676887 16159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:40:58.692725 16164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:58.692698 16165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:40:58.692858 16167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:40:58.699529 16159 server_base.cc:1034] running on GCE node
I20260926 03:40:58.700367 16159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:40:58.701576 16159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:40:58.702937 16159 hybrid_clock.cc:648] HybridClock initialized: now 1790394058702852 us; error 78 us; skew 500 ppm
I20260926 03:40:58.703349 16159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:40:58 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:40:58.707192 16159 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:40:58.709138 16159 webserver.cc:460] Webserver started at http://127.1.221.131:35327/ using document root <none> and password file <none>
I20260926 03:40:58.709969 16159 fs_manager.cc:359] Metadata directory not provided
I20260926 03:40:58.710095 16159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:40:58.710403 16159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:40:58.713768 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "abccda8aa8d64241999e46b38546dcab"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "c2fbeebff726e9ffbec9a5b2ebf5999e"
server_key_iv: "6a2938149156fbfb31417e4370b0a647"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.714596 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "abccda8aa8d64241999e46b38546dcab"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "c2fbeebff726e9ffbec9a5b2ebf5999e"
server_key_iv: "6a2938149156fbfb31417e4370b0a647"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.721015 16159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:40:58.725270 16174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:58.727048 16159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:40:58.727274 16159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "abccda8aa8d64241999e46b38546dcab"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "c2fbeebff726e9ffbec9a5b2ebf5999e"
server_key_iv: "6a2938149156fbfb31417e4370b0a647"
server_key_version: "encryptionkey@0"
I20260926 03:40:58.727546 16159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:40:58.754209 16159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:40:58.759262 16159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:40:58.759608 16159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:40:58.760818 16159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:40:58.762466 16159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:40:58.762547 16159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:58.762648 16159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:40:58.762713 16159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:40:58.799932 16159 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:45169
I20260926 03:40:58.800015 16286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:45169 every 8 connection(s)
I20260926 03:40:58.801414 16159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:40:58.807677 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16159
I20260926 03:40:58.807859 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:40:58.808378 1910 external_mini_cluster.cc:1428] Setting key e8d1c495dd0cc3d594e38f98c1dfb3b4
Sep 26 03:40:58 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:40:58.827610 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.804843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60521 (local address 127.1.221.190:34117)
0926 03:40:58.805399 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.805405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.806608 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 03:40:58.809286 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.809293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.809411 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.809573 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:58.812552 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.813747 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.815964 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.816423 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.816550 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.820593 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:58.820598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:58.820602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:58.820655 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:58.823597 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.824744 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.824749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.824751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.824930 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.825512 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.825518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.825522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.825943 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:58.826136 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.826999 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.827209 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
I20260926 03:40:58.828622 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.805369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:60521)
0926 03:40:58.806321 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:58.806373 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:58.808806 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:58.809944 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:58.809959 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:58.811019 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:40:58.812261 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.812292 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.814005 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.814013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.815738 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.815763 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.816683 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.816689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.816929 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.817730 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:40:58.817754 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:40:58.820303 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:40:58.823815 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:58.823826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:58.823848 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:58.824593 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:58.825051 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:40:58.825055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:40:58.825059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:40:58.825352 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:40:58.826161 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:40:58.826171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:40:58.826575 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 03:40:58.826848 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 03:40:58.827311 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":363,"threads_started":1}
I20260926 03:40:58.830219 16287 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:40:58.830693 16287 heartbeater.cc:464] Registering TS with master...
I20260926 03:40:58.831708 16287 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
I20260926 03:40:58.833570 15830 ts_manager.cc:194] Registered new tserver with Master: abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169)
I20260926 03:40:58.834486 15830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:60521
I20260926 03:40:58.838536 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:40:58.857857 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.843196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.1.221.190:34117)
0926 03:40:58.843453 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.843458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.843930 (+ 472us) server_negotiation.cc:407] Connection header received
0926 03:40:58.844096 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.844101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.844200 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.844310 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:58.845775 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.846570 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.848175 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.848571 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.848683 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.851894 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:58.851900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:58.851904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:58.851950 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:58.854823 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.855838 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.855842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.855844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.856012 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:58.856381 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:58.856384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:58.856387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:58.856749 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:58.856960 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.857384 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.857598 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:40:58.862481 15830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:40:58.869659 15830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:40:58.896704 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.885606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.1.221.131:45169)
0926 03:40:58.886451 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.886459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.886498 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:40:58.886810 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.886819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.887298 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.887645 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:58.888585 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.890251 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.893207 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.894810 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.894935 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.895219 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.896045 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.896276 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":452,"threads_started":1}
I20260926 03:40:58.898118 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.887175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.1.221.129:43387)
0926 03:40:58.888586 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.888595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.888625 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:40:58.888739 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.888747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.889054 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.889267 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:58.890305 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.891932 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.894607 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.896121 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.896249 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.896492 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.897644 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.897794 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":999,"threads_started":1}
I20260926 03:40:58.898765 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.885179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45169 (local address 127.0.0.1:54992)
0926 03:40:58.886255 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:58.886305 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:58.886494 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:58.887851 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:58.887861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:58.887894 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:58.888341 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.888355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.890434 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.890439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.893020 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.893036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.895463 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.895470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.895721 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.895784 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 03:40:58.897077 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0926 03:40:58.897331 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":672,"threads_started":1}
I20260926 03:40:58.898797 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.886763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:57822)
0926 03:40:58.887498 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:58.887523 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:58.887652 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:58.889561 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:58.889566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:58.889593 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:58.890089 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.890102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.892107 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.892112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.894410 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.894425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.896662 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.896668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.897200 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.897232 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:40:58.897321 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:40:58.897459 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":322,"threads_started":1}
I20260926 03:40:58.903986 15954 tablet_service.cc:1467] Processing CreateTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 (DEFAULT_TABLE table=test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:58.904412 16222 tablet_service.cc:1467] Processing CreateTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 (DEFAULT_TABLE table=test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:58.905604 15954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d7ad99be974e8a84ca24c8c01737c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:58.906170 16222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d7ad99be974e8a84ca24c8c01737c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:58.909022 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.888637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:46399 (local address 127.0.0.1:41010)
0926 03:40:58.898695 (+ 10058us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:58.899179 (+ 484us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:58.899313 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:58.900162 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:58.900167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:58.900197 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:58.900568 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.900579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.902735 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.902740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.904951 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:58.904971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.908105 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.908112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:58.908317 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.908370 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:40:58.908479 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:40:58.908654 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9878,"thread_start_us":8804,"threads_started":1}
I20260926 03:40:58.909479 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:58.887758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.1.221.130:46399)
0926 03:40:58.897337 (+ 9579us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:58.897363 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:58.899309 (+ 1946us) server_negotiation.cc:407] Connection header received
0926 03:40:58.899555 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:58.899564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:58.899929 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:58.900162 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:58.900821 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.902531 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.905198 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:58.907549 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:58.907703 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:58.907927 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:58.908605 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 03:40:58.908852 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9394,"thread_start_us":9264,"threads_started":1}
I20260926 03:40:58.913699 16088 tablet_service.cc:1467] Processing CreateTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 (DEFAULT_TABLE table=test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:40:58.915418 16088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d7ad99be974e8a84ca24c8c01737c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:40:58.922164 16306 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Bootstrap starting.
I20260926 03:40:58.927152 16306 tablet_bootstrap.cc:654] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:58.928552 16307 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Bootstrap starting.
I20260926 03:40:58.928812 16306 log.cc:824] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:58.933272 16307 tablet_bootstrap.cc:654] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:58.933656 16306 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: No bootstrap required, opened a new log
I20260926 03:40:58.934022 16306 ts_tablet_manager.cc:1397] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 03:40:58.934109 16309 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Bootstrap starting.
I20260926 03:40:58.935135 16307 log.cc:824] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:58.937952 16309 tablet_bootstrap.cc:654] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Neither blocks nor log segments found. Creating new log.
I20260926 03:40:58.939123 16307 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: No bootstrap required, opened a new log
I20260926 03:40:58.939137 16309 log.cc:824] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Log is configured to *not* fsync() on all Append() calls
I20260926 03:40:58.939502 16307 ts_tablet_manager.cc:1397] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 03:40:58.943686 16309 tablet_bootstrap.cc:492] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: No bootstrap required, opened a new log
I20260926 03:40:58.944023 16309 ts_tablet_manager.cc:1397] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:40:58.945580 16306 raft_consensus.cc:348] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.946183 16306 raft_consensus.cc:374] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:58.946291 16306 raft_consensus.cc:729] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abccda8aa8d64241999e46b38546dcab, State: Initialized, Role: FOLLOWER
I20260926 03:40:58.947114 16306 consensus_queue.cc:260] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.948681 16287 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
I20260926 03:40:58.949425 16306 ts_tablet_manager.cc:1428] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260926 03:40:58.950675 16307 raft_consensus.cc:348] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.951156 16307 raft_consensus.cc:374] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:58.951259 16307 raft_consensus.cc:729] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277941c72bff47e9a2a26623a3cac6a1, State: Initialized, Role: FOLLOWER
I20260926 03:40:58.952018 16307 consensus_queue.cc:260] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.956888 16307 ts_tablet_manager.cc:1428] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260926 03:40:58.956678 16309 raft_consensus.cc:348] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.957268 16309 raft_consensus.cc:374] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:40:58.957383 16309 raft_consensus.cc:729] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd490db62ed4935a42f8ae3674c6ced, State: Initialized, Role: FOLLOWER
I20260926 03:40:58.958216 16309 consensus_queue.cc:260] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:58.959414 16153 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
I20260926 03:40:58.960276 16309 ts_tablet_manager.cc:1428] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260926 03:40:59.053577 16288 tablet.cc:2369] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:40:59.058723 16020 tablet.cc:2369] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:59.132905 16314 raft_consensus.cc:482] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:40:59.133199 16314 raft_consensus.cc:504] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:59.134985 16314 leader_election.cc:288] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169), 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387)
I20260926 03:40:59.144531 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.136519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:46073 (local address 127.1.221.129:43387)
0926 03:40:59.136794 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.136799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.137219 (+ 420us) server_negotiation.cc:407] Connection header received
0926 03:40:59.137408 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.137413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.137494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.137599 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:59.138261 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.139349 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.142121 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.143159 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.143332 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.143444 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.144086 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.144254 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:40:59.144608 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.135520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34951 (local address 127.1.221.131:45169)
0926 03:40:59.135779 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.135787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.136394 (+ 607us) server_negotiation.cc:407] Connection header received
0926 03:40:59.136763 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.136769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.136859 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.136917 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:40:59.137949 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.139110 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.142043 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.143202 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.143404 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.143499 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.144127 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.144339 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:40:59.144728 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.136250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.1.221.130:46073)
0926 03:40:59.137089 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:59.137107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:59.137260 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:59.137646 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:59.137649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:59.137675 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:59.138096 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:59.138106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.139519 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.139524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:59.141832 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:59.141851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.143642 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.143650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:59.143873 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.143924 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:40:59.144013 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:40:59.144172 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":81,"thread_start_us":532,"threads_started":1}
I20260926 03:40:59.144783 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.135422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45169 (local address 127.1.221.130:34951)
0926 03:40:59.136237 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:40:59.136263 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:40:59.136553 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:40:59.137121 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:40:59.137124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:40:59.137147 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:40:59.137798 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:59.137812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.139263 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.139268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:59.141839 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:40:59.141851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.143677 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.143683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:40:59.143878 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.143936 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:40:59.144030 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:40:59.144206 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":128,"spinlock_wait_cycles":36736,"thread_start_us":413,"threads_started":1}
I20260926 03:40:59.145994 16242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d7ad99be974e8a84ca24c8c01737c5" candidate_uuid: "9fd490db62ed4935a42f8ae3674c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abccda8aa8d64241999e46b38546dcab" is_pre_election: true
I20260926 03:40:59.146065 15974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d7ad99be974e8a84ca24c8c01737c5" candidate_uuid: "9fd490db62ed4935a42f8ae3674c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277941c72bff47e9a2a26623a3cac6a1" is_pre_election: true
I20260926 03:40:59.146534 16242 raft_consensus.cc:2454] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd490db62ed4935a42f8ae3674c6ced in term 0.
I20260926 03:40:59.146642 15974 raft_consensus.cc:2454] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd490db62ed4935a42f8ae3674c6ced in term 0.
I20260926 03:40:59.147275 16044 leader_election.cc:302] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd490db62ed4935a42f8ae3674c6ced, abccda8aa8d64241999e46b38546dcab; no voters:
I20260926 03:40:59.147760 16314 raft_consensus.cc:2781] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:40:59.147940 16314 raft_consensus.cc:482] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:40:59.148064 16314 raft_consensus.cc:3037] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:59.150679 16314 raft_consensus.cc:504] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:59.151432 16314 leader_election.cc:288] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [CANDIDATE]: Term 1 election: Requested vote from peers abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169), 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387)
I20260926 03:40:59.152025 16242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d7ad99be974e8a84ca24c8c01737c5" candidate_uuid: "9fd490db62ed4935a42f8ae3674c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abccda8aa8d64241999e46b38546dcab"
I20260926 03:40:59.152109 15974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d7ad99be974e8a84ca24c8c01737c5" candidate_uuid: "9fd490db62ed4935a42f8ae3674c6ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277941c72bff47e9a2a26623a3cac6a1"
I20260926 03:40:59.152252 16242 raft_consensus.cc:3037] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:59.152323 15974 raft_consensus.cc:3037] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:40:59.154863 15974 raft_consensus.cc:2454] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd490db62ed4935a42f8ae3674c6ced in term 1.
I20260926 03:40:59.155037 16242 raft_consensus.cc:2454] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd490db62ed4935a42f8ae3674c6ced in term 1.
I20260926 03:40:59.155336 16044 leader_election.cc:302] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277941c72bff47e9a2a26623a3cac6a1, 9fd490db62ed4935a42f8ae3674c6ced; no voters:
I20260926 03:40:59.155685 16314 raft_consensus.cc:2781] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:40:59.156210 16314 raft_consensus.cc:686] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 LEADER]: Becoming Leader. State: Replica: 9fd490db62ed4935a42f8ae3674c6ced, State: Running, Role: LEADER
I20260926 03:40:59.156772 16314 consensus_queue.cc:237] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } }
I20260926 03:40:59.162473 15829 catalog_manager.cc:5495] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130). New cstate: current_term: 1 leader_uuid: "9fd490db62ed4935a42f8ae3674c6ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } health_report { overall_health: HEALTHY } } }
W20260926 03:40:59.189388 16154 tablet.cc:2369] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:40:59.215591 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.209513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.1.221.130:46399)
0926 03:40:59.209768 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.209775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.209847 (+ 72us) server_negotiation.cc:407] Connection header received
0926 03:40:59.210097 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.210101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.210191 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.210327 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:40:59.210713 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.211542 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.213037 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.213591 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.213809 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.214378 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:40:59.214942 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:40:59.215085 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.215191 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.215282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 03:40:59.226192 15974 raft_consensus.cc:1261] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Refusing update from remote peer 9fd490db62ed4935a42f8ae3674c6ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:59.226418 16242 raft_consensus.cc:1261] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Refusing update from remote peer 9fd490db62ed4935a42f8ae3674c6ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:40:59.227569 16317 consensus_queue.cc:1035] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:59.228060 16314 consensus_queue.cc:1035] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:40:59.242076 16320 mvcc.cc:205] Tried to move back new op lower bound from 7333454066577358848 to 7333454066310610944. Current Snapshot: MvccSnapshot[applied={T|T < 7333454066577358848}]
I20260926 03:40:59.260803 15829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:40:59.261171 15829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:40:59.265997 15829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Sending DeleteTablet for 3 replicas of tablet c6d7ad99be974e8a84ca24c8c01737c5
I20260926 03:40:59.267938 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15891
I20260926 03:40:59.267967 15954 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:59 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57822
I20260926 03:40:59.268023 16222 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:59 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54992
I20260926 03:40:59.268883 16088 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d7ad99be974e8a84ca24c8c01737c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:40:59 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41010
I20260926 03:40:59.269127 16329 tablet_replica.cc:330] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: stopping tablet replica
I20260926 03:40:59.269585 16329 raft_consensus.cc:2229] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:40:59.270190 16329 raft_consensus.cc:2258] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:59.272855 16329 ts_tablet_manager.cc:1905] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:40:59.280040 16332 tablet_replica.cc:330] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: stopping tablet replica
I20260926 03:40:59.280840 16332 raft_consensus.cc:2229] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:40:59.281872 16332 raft_consensus.cc:2258] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:59.281934 16329 ts_tablet_manager.cc:1918] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:40:59.282109 16329 log.cc:1196] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/c6d7ad99be974e8a84ca24c8c01737c5
I20260926 03:40:59.282727 16329 ts_tablet_manager.cc:1939] T c6d7ad99be974e8a84ca24c8c01737c5 P abccda8aa8d64241999e46b38546dcab: Deleting consensus metadata
I20260926 03:40:59.284986 16332 ts_tablet_manager.cc:1905] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:40:59.284951 15817 catalog_manager.cc:4841] TS abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169): tablet c6d7ad99be974e8a84ca24c8c01737c5 (table test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]) successfully deleted
I20260926 03:40:59.286931 16330 tablet_replica.cc:330] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: stopping tablet replica
I20260926 03:40:59.287559 16330 raft_consensus.cc:2229] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:40:59.288276 16330 raft_consensus.cc:2258] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:40:59.290777 16330 ts_tablet_manager.cc:1905] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:40:59.293833 16332 ts_tablet_manager.cc:1918] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:40:59.294160 16332 log.cc:1196] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/c6d7ad99be974e8a84ca24c8c01737c5
I20260926 03:40:59.294843 16332 ts_tablet_manager.cc:1939] T c6d7ad99be974e8a84ca24c8c01737c5 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting consensus metadata
I20260926 03:40:59.297070 15817 catalog_manager.cc:4841] TS 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399): tablet c6d7ad99be974e8a84ca24c8c01737c5 (table test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]) successfully deleted
I20260926 03:40:59.297995 16330 ts_tablet_manager.cc:1918] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:40:59.298177 16330 log.cc:1196] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/c6d7ad99be974e8a84ca24c8c01737c5
I20260926 03:40:59.298767 16330 ts_tablet_manager.cc:1939] T c6d7ad99be974e8a84ca24c8c01737c5 P 277941c72bff47e9a2a26623a3cac6a1: Deleting consensus metadata
Sep 26 03:40:59 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:40:59.300565 15817 catalog_manager.cc:4841] TS 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387): tablet c6d7ad99be974e8a84ca24c8c01737c5 (table test-table [id=ec37c24e5f234f09a33b2a04a3356ba0]) successfully deleted
I20260926 03:40:59.307590 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.269545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.1.221.129:43387)
0926 03:40:59.269761 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.269769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.286145 (+ 16376us) server_negotiation.cc:407] Connection header received
0926 03:40:59.286369 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.286404 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.286544 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.286747 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:59.289823 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.291139 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.293321 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.294070 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.294186 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.300652 (+ 6466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:59.300668 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:59.300689 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:59.300754 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:59.304097 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.305215 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.305223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.305226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.305406 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.305905 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.305909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.305911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.306323 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:59.306555 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.307019 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.307220 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:40:59.308382 16014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:59.406639 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15891
I20260926 03:40:59.423758 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16025
Sep 26 03:40:59 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:40:59.445115 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.424823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.1.221.130:46399)
0926 03:40:59.424993 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.424998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.425020 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:40:59.425161 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.425165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.425239 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.425335 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:59.427866 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.428959 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.431332 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.432091 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.432260 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.438396 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:59.438417 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:59.438435 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:59.438480 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:59.441973 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.442845 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.442853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.442855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.443009 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.443523 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.443526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.443528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.443908 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:59.444270 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.444521 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.444752 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:40:59.445873 16148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:59.566103 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16025
I20260926 03:40:59.585433 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16159
Sep 26 03:40:59 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:40:59.608929 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:40:59.586382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.1.221.131:45169)
0926 03:40:59.586651 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 03:40:59.586656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:40:59.586676 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:40:59.586740 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:40:59.586744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:40:59.586813 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:40:59.586900 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:40:59.589162 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.589985 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.591455 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:40:59.591893 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:40:59.592017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:40:59.602510 (+ 10493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:40:59.602530 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:40:59.602547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:40:59.602597 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:40:59.605897 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.606749 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.606757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.606760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.606906 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:40:59.607276 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:40:59.607279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:40:59.607281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:40:59.607717 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:40:59.607998 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 03:40:59.608320 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 03:40:59.608567 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 03:40:59.609579 16282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:59.709944 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16159
I20260926 03:40:59.725320 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 15798
I20260926 03:40:59.726724 15860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:40:59.840183 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 15798
I20260926 03:40:59.863839 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--webserver_interface=127.1.221.190
--webserver_port=36081
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:00.277984 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:00.278416 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:00.278616 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:00.287147 16337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:41:00.287309 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:00.287370 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:00.287498 16337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:41:00.287568 16337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:41:00.298885 16337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=36081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:00.300725 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:00.302474 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:00.330533 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:00.330646 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:00.330556 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:00.331423 16337 server_base.cc:1034] running on GCE node
I20260926 03:41:00.332587 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:00.334704 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:00.335973 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790394060335902 us; error 57 us; skew 500 ppm
I20260926 03:41:00.336499 16337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:00 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:00.340644 16337 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:41:00.342839 16337 webserver.cc:460] Webserver started at http://127.1.221.190:36081/ using document root <none> and password file <none>
I20260926 03:41:00.343972 16337 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:00.344162 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:00.352095 16337 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:41:00.356207 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:00.358353 16337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:41:00.358623 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "5162b4a1bb5248aaa9d5615d962715e6"
format_stamp: "Formatted at 2026-09-26 03:40:56 on dist-test-slave-lqrk"
server_key: "b3a9423ae6ebcfbc0abbaa93307cb1ee"
server_key_iv: "fee04a67e9699e645471cacbd201e67d"
server_key_version: "encryptionkey@0"
I20260926 03:41:00.360004 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:00.386736 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:00.391894 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:00.392220 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:00.414548 16337 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34117
I20260926 03:41:00.414608 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34117 every 8 connection(s)
I20260926 03:41:00.416328 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:41:00.425195 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16337
I20260926 03:41:00.426455 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:43387
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=32929
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:41:00.426801 16405 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:41:00.432431 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap starting.
I20260926 03:41:00.459026 16405 log.cc:824] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:00.474527 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:41:00.475165 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap complete.
I20260926 03:41:00.485694 16405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:00.486780 16405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Initialized, Role: FOLLOWER
I20260926 03:41:00.487637 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:00.487913 16405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:41:00.488047 16405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:41:00.488231 16405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:41:00.493144 16405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:00.493897 16405 leader_election.cc:302] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5162b4a1bb5248aaa9d5615d962715e6; no voters:
I20260926 03:41:00.494625 16405 leader_election.cc:288] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:41:00.494797 16409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:41:00.496135 16409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 LEADER]: Becoming Leader. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Running, Role: LEADER
I20260926 03:41:00.496985 16409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:00.497661 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:41:00.500125 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5162b4a1bb5248aaa9d5615d962715e6. Latest consensus state: current_term: 2 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:41:00.500699 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:00.501421 16410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:41:00.501767 16410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:00.512487 16415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:41:00.523413 16415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:41:00.525456 16415 catalog_manager.cc:1259] Loaded cluster ID: 33460f8e96344dd6bea74971afd8857f
I20260926 03:41:00.525697 16415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:41:00.530195 16415 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 03:41:00.550395 16415 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 03:41:00.560362 16415 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:41:00.562501 16415 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Loaded TSK: 0
I20260926 03:41:00.564286 16415 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:41:00.844859 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:00.845178 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:00.845304 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:00.853168 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:00.853302 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:00.853456 16407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:41:00.864327 16407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:43387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=32929
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:00.865811 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:00.867275 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:00.884045 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:00.884044 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:00.884548 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:00.884972 16407 server_base.cc:1034] running on GCE node
I20260926 03:41:00.885852 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:00.887012 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:00.888253 16407 hybrid_clock.cc:648] HybridClock initialized: now 1790394060888177 us; error 76 us; skew 500 ppm
I20260926 03:41:00.888622 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:00 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:00.892063 16407 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:41:00.893793 16407 webserver.cc:460] Webserver started at http://127.1.221.129:32929/ using document root <none> and password file <none>
I20260926 03:41:00.894559 16407 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:00.894695 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:00.901250 16407 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:41:00.904647 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:00.906414 16407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:41:00.906620 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "277941c72bff47e9a2a26623a3cac6a1"
format_stamp: "Formatted at 2026-09-26 03:40:57 on dist-test-slave-lqrk"
server_key: "15b9182a15349f3772c1ec2dabc5de98"
server_key_iv: "e0a88f8eb4b139531d732e2a4c74f511"
server_key_version: "encryptionkey@0"
I20260926 03:41:00.907879 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:00.926913 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:00.931950 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:00.932272 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:00.933456 16407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:00.935093 16407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:00.935225 16407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:00.935354 16407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:00.935470 16407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:41:00.972071 16407 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43387
I20260926 03:41:00.972131 16554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43387 every 8 connection(s)
I20260926 03:41:00.973637 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:41:00.979699 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16407
I20260926 03:41:00.980417 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:46399
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=33281
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:41:00 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:01.002526 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:00.977200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:49535)
0926 03:41:00.978076 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:00.978132 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:00.980074 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:00.982423 (+ 2349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:00.982433 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:00.983083 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:00.983747 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:00.983769 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:00.986406 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:00.986411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:00.987899 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:00.987922 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:00.989368 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:00.989378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:00.989708 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:00.990532 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:00.990564 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:00.993360 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:00.997015 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:00.997025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:00.997039 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:00.997563 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:00.998060 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:00.998066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:00.998070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:00.998303 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:00.999177 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:00.999185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:00.999864 (+ 679us) client_negotiation.cc:769] Sending connection context
0926 03:41:01.000224 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 03:41:01.000920 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":361,"threads_started":1}
I20260926 03:41:01.002526 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:00.976945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49535 (local address 127.1.221.190:34117)
0926 03:41:00.977562 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:00.977576 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:00.978295 (+ 719us) server_negotiation.cc:407] Connection header received
0926 03:41:00.980957 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:00.980986 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:00.981669 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:00.982239 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:00.984041 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:00.986109 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:00.988123 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:00.989029 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:00.989204 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:00.993731 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:00.993741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:00.993758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:00.993806 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:00.996804 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:00.997721 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:00.997730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:00.997736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:00.997936 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:00.998426 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:00.998430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:00.998432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:00.999023 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:00.999225 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:01.000543 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0926 03:41:01.000998 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":284,"threads_started":1}
I20260926 03:41:01.005996 16555 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:41:01.006358 16555 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:01.007236 16555 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
I20260926 03:41:01.010128 16370 ts_manager.cc:194] Registered new tserver with Master: 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387)
I20260926 03:41:01.012235 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:49535
W20260926 03:41:01.326310 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:01.326608 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:01.326730 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:01.334318 16559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:01.334451 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:01.334594 16559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:41:01.345011 16559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:46399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=33281
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:01.346397 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:01.347751 16559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:01.360733 16564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:01.360827 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:01.361197 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:01.361737 16559 server_base.cc:1034] running on GCE node
I20260926 03:41:01.362499 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:01.363674 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:01.364895 16559 hybrid_clock.cc:648] HybridClock initialized: now 1790394061364853 us; error 34 us; skew 500 ppm
I20260926 03:41:01.365290 16559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:01 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:01.368686 16559 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:41:01.370288 16559 webserver.cc:460] Webserver started at http://127.1.221.130:33281/ using document root <none> and password file <none>
I20260926 03:41:01.371039 16559 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:01.371176 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:01.377523 16559 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 03:41:01.380769 16574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.382393 16559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:41:01.382587 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "9fd490db62ed4935a42f8ae3674c6ced"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "74a93cd8f4ede0d6a569c821061efa26"
server_key_iv: "a6480759008a6dbd1d880654f17ddace"
server_key_version: "encryptionkey@0"
I20260926 03:41:01.383683 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:01.416164 16559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:01.420722 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:01.421005 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:01.422076 16559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:01.423579 16559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:01.423703 16559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.423870 16559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:01.423952 16559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.457233 16559 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:46399
I20260926 03:41:01.457331 16686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:46399 every 8 connection(s)
I20260926 03:41:01.459266 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:41:01.467878 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16559
I20260926 03:41:01.468665 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:45169
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=35327
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:41:01 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:01.481755 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:01.462367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:33087 (local address 127.1.221.190:34117)
0926 03:41:01.462558 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:01.462563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:01.464219 (+ 1656us) server_negotiation.cc:407] Connection header received
0926 03:41:01.466369 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:01.466374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:01.466504 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:01.466598 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:01.468728 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.469557 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.471259 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.471752 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.471871 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:01.475588 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:01.475593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:01.475596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:01.475632 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:01.478297 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.479122 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.479126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.479129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:01.479272 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.479755 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.479759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.479761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:01.480050 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:01.480209 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:01.481111 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 03:41:01.481401 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:41:01.484284 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:01.463064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:33087)
0926 03:41:01.464037 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:01.464088 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:01.466043 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:01.466888 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:01.466898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:01.467810 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:01.468516 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:01.468536 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.469762 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.469766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:01.471082 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:01.471096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.471997 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.472005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:01.472244 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:01.472903 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:01.472929 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:01.475245 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:01.478468 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:01.478479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:01.478499 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:01.478984 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:01.479409 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:01.479414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:01.479417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:01.479626 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:01.480184 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:01.480192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:01.480645 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 03:41:01.482062 (+ 1417us) client_negotiation.cc:240] Negotiation successful
0926 03:41:01.482729 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":440,"threads_started":1}
I20260926 03:41:01.486200 16687 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:41:01.486538 16687 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:01.487506 16687 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
I20260926 03:41:01.489352 16370 ts_manager.cc:194] Registered new tserver with Master: 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399)
I20260926 03:41:01.490468 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:33087
W20260926 03:41:01.822216 16690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:01.822563 16690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:01.822667 16690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:01.831368 16690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:01.831559 16690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:01.831681 16690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:41:01.842598 16690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:45169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=35327
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:01.844066 16690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:01.845760 16690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:01.861208 16698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:01.861529 16696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:01.861245 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:01.862089 16690 server_base.cc:1034] running on GCE node
I20260926 03:41:01.862928 16690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:01.864384 16690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:01.865653 16690 hybrid_clock.cc:648] HybridClock initialized: now 1790394061865581 us; error 53 us; skew 500 ppm
I20260926 03:41:01.866142 16690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:01 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:01.870780 16690 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:41:01.872962 16690 webserver.cc:460] Webserver started at http://127.1.221.131:35327/ using document root <none> and password file <none>
I20260926 03:41:01.873967 16690 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:01.874140 16690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:01.881779 16690 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:41:01.886132 16705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.888535 16690 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 03:41:01.888809 16690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "abccda8aa8d64241999e46b38546dcab"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "c2fbeebff726e9ffbec9a5b2ebf5999e"
server_key_iv: "6a2938149156fbfb31417e4370b0a647"
server_key_version: "encryptionkey@0"
I20260926 03:41:01.890188 16690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:01.922926 16690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:01.928997 16690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:01.929334 16690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:01.930629 16690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:01.932461 16690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:01.932579 16690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.932703 16690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:01.932775 16690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:01.970727 16690 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:45169
I20260926 03:41:01.970793 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:45169 every 8 connection(s)
I20260926 03:41:01.972934 16690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:41:01.975818 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16690
Sep 26 03:41:01 dist-test-slave-lqrk krb5kdc[15785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:02.000345 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:01.976792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.1.221.190:34117)
0926 03:41:01.983398 (+ 6606us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:01.983404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:01.983988 (+ 584us) server_negotiation.cc:407] Connection header received
0926 03:41:01.984161 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:01.984167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:01.984271 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:01.984439 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:01.986577 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.987828 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.989604 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.990242 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.990393 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:01.993289 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:01.993294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:01.993298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:01.993338 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:01.996740 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.997813 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.997817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.997819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:01.998113 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.998550 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.998555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.998558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:01.998960 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:01.999212 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:01.999740 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.000004 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6452,"thread_start_us":6302,"threads_started":1}
I20260926 03:41:02.002312 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:01.976402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35507 (local address 127.1.221.190:34117)
0926 03:41:01.976601 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:01.976607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:01.980449 (+ 3842us) server_negotiation.cc:407] Connection header received
0926 03:41:01.982893 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:01.982900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:01.983029 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:01.983145 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:01.985730 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.986862 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.989048 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.989714 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.989825 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:01.994211 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:01.994220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:01.994223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:01.994279 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:01.997999 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.999161 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.999166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.999169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:01.999348 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:01.999958 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:01.999964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:01.999967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.000379 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:02.000540 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.001679 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.001932 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:41:02.003434 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:01.976891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:35507)
0926 03:41:01.980181 (+ 3290us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:01.980245 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:01.982456 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:01.983825 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:01.983843 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:01.984757 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:01.985467 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:01.985491 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.987095 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.987102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:01.988853 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:01.988868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:01.989996 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:01.990004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:01.990326 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:01.991175 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:01.991201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:01.993894 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:01.998246 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:01.998257 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:01.998275 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:01.998927 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:01.999512 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:01.999516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:01.999518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:01.999790 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:02.000529 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:02.000539 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:02.001091 (+ 552us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.001455 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.002030 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2892,"thread_start_us":2610,"threads_started":1}
I20260926 03:41:02.005261 16818 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34117
I20260926 03:41:02.005738 16818 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:02.006690 16818 heartbeater.cc:511] Master 127.1.221.190:34117 requested a full tablet report, sending...
I20260926 03:41:02.008630 16370 ts_manager.cc:194] Registered new tserver with Master: abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169)
I20260926 03:41:02.009645 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:35507
I20260926 03:41:02.015897 16555 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
I20260926 03:41:02.017776 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:41:02.026443 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.018966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.1.221.190:34117)
0926 03:41:02.019156 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.019162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.019681 (+ 519us) server_negotiation.cc:407] Connection header received
0926 03:41:02.019916 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.019921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.020115 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.020294 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:41:02.020844 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.021747 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.023718 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.024277 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.024413 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.025094 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:41:02.025803 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:41:02.025946 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.026074 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.026182 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:41:02.027807 16369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:41:02.036043 16369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:41:02.066080 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.052076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.1.221.130:46399)
0926 03:41:02.056251 (+ 4175us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.056262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.056302 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:41:02.056451 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.056462 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.056798 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.057035 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:02.058352 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.060020 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.063321 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.064576 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.064714 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.065023 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.065446 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.065634 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3280,"thread_start_us":3113,"threads_started":1}
I20260926 03:41:02.066176 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.051707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:46399 (local address 127.0.0.1:44836)
0926 03:41:02.052673 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.052714 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:02.052912 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:02.057371 (+ 4459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:02.057378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:02.057438 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:02.058088 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.058108 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.060209 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.060215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.063093 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.063111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.064884 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.064891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.065187 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.065233 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.065378 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.065697 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":143,"thread_start_us":451,"threads_started":1}
I20260926 03:41:02.068166 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.053611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.1.221.131:45169)
0926 03:41:02.056143 (+ 2532us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.056153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.056196 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:41:02.056398 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.056406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.056787 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.058147 (+ 1360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:02.058512 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.060198 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.064568 (+ 4370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.066345 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.066514 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.067245 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.067356 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.067649 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2317,"thread_start_us":2115,"threads_started":1}
I20260926 03:41:02.068984 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.053090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45169 (local address 127.0.0.1:57890)
0926 03:41:02.055953 (+ 2863us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.055992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:02.056197 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:02.057108 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:02.057117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:02.057154 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:02.057880 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.057896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.060359 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.060364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.064235 (+ 3871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.064252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.066634 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.066640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.066816 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.066851 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.066941 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.068546 (+ 1605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2690,"mutex_wait_us":116,"thread_start_us":2516,"threads_started":1}
I20260926 03:41:02.070713 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.056976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38318)
0926 03:41:02.059891 (+ 2915us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.059913 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:02.060059 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:02.060803 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:02.060808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:02.060837 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:02.061669 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.061685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.064649 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.064654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.067005 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.067113 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.069944 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.069953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.070117 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.070144 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.070232 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.070368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"mutex_wait_us":180,"thread_start_us":2592,"threads_started":1}
I20260926 03:41:02.071014 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.056302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.1.221.129:43387)
0926 03:41:02.059311 (+ 3009us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.059325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.060069 (+ 744us) server_negotiation.cc:407] Connection header received
0926 03:41:02.060214 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.060223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.060553 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.060790 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:02.062348 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.064374 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.067457 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.069333 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.069469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.069715 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.070397 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.070654 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
I20260926 03:41:02.071301 16622 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca7205460346f389c9835da42aaa86 (DEFAULT_TABLE table=test-table [id=e4751132b34440adaf7759b80fddc2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:02.074913 16490 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca7205460346f389c9835da42aaa86 (DEFAULT_TABLE table=test-table [id=e4751132b34440adaf7759b80fddc2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:02.075582 16753 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca7205460346f389c9835da42aaa86 (DEFAULT_TABLE table=test-table [id=e4751132b34440adaf7759b80fddc2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:02.076622 16490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca7205460346f389c9835da42aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:02.077181 16753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca7205460346f389c9835da42aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:02.080879 16622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca7205460346f389c9835da42aaa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:02.092849 16834 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Bootstrap starting.
I20260926 03:41:02.097497 16834 tablet_bootstrap.cc:654] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:02.099807 16834 log.cc:824] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:02.101316 16835 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Bootstrap starting.
I20260926 03:41:02.103648 16836 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Bootstrap starting.
I20260926 03:41:02.105193 16834 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: No bootstrap required, opened a new log
I20260926 03:41:02.105544 16834 ts_tablet_manager.cc:1397] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260926 03:41:02.107244 16836 tablet_bootstrap.cc:654] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:02.107255 16835 tablet_bootstrap.cc:654] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:02.109062 16836 log.cc:824] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:02.109395 16835 log.cc:824] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:02.113117 16836 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: No bootstrap required, opened a new log
I20260926 03:41:02.113466 16836 ts_tablet_manager.cc:1397] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:41:02.113772 16835 tablet_bootstrap.cc:492] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: No bootstrap required, opened a new log
I20260926 03:41:02.114145 16835 ts_tablet_manager.cc:1397] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260926 03:41:02.116947 16834 raft_consensus.cc:348] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.117592 16834 raft_consensus.cc:374] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:02.117722 16834 raft_consensus.cc:729] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abccda8aa8d64241999e46b38546dcab, State: Initialized, Role: FOLLOWER
I20260926 03:41:02.118642 16834 consensus_queue.cc:260] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.120191 16818 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
I20260926 03:41:02.121601 16836 raft_consensus.cc:348] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.122123 16836 raft_consensus.cc:374] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:02.122262 16836 raft_consensus.cc:729] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd490db62ed4935a42f8ae3674c6ced, State: Initialized, Role: FOLLOWER
I20260926 03:41:02.123607 16836 consensus_queue.cc:260] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.124426 16834 ts_tablet_manager.cc:1428] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Time spent starting tablet: real 0.019s user 0.012s sys 0.003s
I20260926 03:41:02.125415 16835 raft_consensus.cc:348] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.125916 16687 heartbeater.cc:503] Master 127.1.221.190:34117 was elected leader, sending a full tablet report...
I20260926 03:41:02.126111 16835 raft_consensus.cc:374] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:02.126258 16835 raft_consensus.cc:729] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277941c72bff47e9a2a26623a3cac6a1, State: Initialized, Role: FOLLOWER
I20260926 03:41:02.126680 16836 ts_tablet_manager.cc:1428] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20260926 03:41:02.127293 16835 consensus_queue.cc:260] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.131016 16835 ts_tablet_manager.cc:1428] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260926 03:41:02.182801 16842 raft_consensus.cc:482] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:41:02.183230 16842 raft_consensus.cc:504] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.185618 16842 leader_election.cc:288] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399), abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169)
I20260926 03:41:02.194068 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.187233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36823 (local address 127.1.221.131:45169)
0926 03:41:02.187507 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.187513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.187944 (+ 431us) server_negotiation.cc:407] Connection header received
0926 03:41:02.188128 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.188134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.188216 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.188353 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:02.189109 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.189937 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.192226 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.193090 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.193278 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.193359 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.193665 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.193807 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:41:02.194089 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.186199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:52847 (local address 127.1.221.130:46399)
0926 03:41:02.186473 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.186481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.187062 (+ 581us) server_negotiation.cc:407] Connection header received
0926 03:41:02.187422 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.187430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.187532 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.187652 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:02.188465 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.189515 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.191894 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.192877 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.193020 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.193089 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.193560 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.193786 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:41:02.194197 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.186035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:46399 (local address 127.1.221.129:52847)
0926 03:41:02.186910 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.186929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:02.187130 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:02.187739 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:02.187744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:02.187783 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:02.188250 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.188265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.189736 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.189743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.191683 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.191702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.193115 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.193121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.193319 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.193368 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.193502 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.193632 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":39,"spinlock_wait_cycles":30464,"thread_start_us":555,"threads_started":1}
I20260926 03:41:02.194312 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.187069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45169 (local address 127.1.221.129:36823)
0926 03:41:02.187812 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.187831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:02.187987 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:02.188437 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:02.188442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:02.188473 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:02.188910 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.188929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.190145 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.190149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.192016 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:02.192035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.193330 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.193335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:02.193484 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.193518 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:41:02.193620 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:41:02.193779 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":73,"thread_start_us":351,"threads_started":1}
I20260926 03:41:02.195570 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca7205460346f389c9835da42aaa86" candidate_uuid: "277941c72bff47e9a2a26623a3cac6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd490db62ed4935a42f8ae3674c6ced" is_pre_election: true
I20260926 03:41:02.195570 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca7205460346f389c9835da42aaa86" candidate_uuid: "277941c72bff47e9a2a26623a3cac6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abccda8aa8d64241999e46b38546dcab" is_pre_election: true
I20260926 03:41:02.196173 16642 raft_consensus.cc:2454] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277941c72bff47e9a2a26623a3cac6a1 in term 0.
I20260926 03:41:02.196173 16773 raft_consensus.cc:2454] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277941c72bff47e9a2a26623a3cac6a1 in term 0.
I20260926 03:41:02.196975 16446 leader_election.cc:302] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277941c72bff47e9a2a26623a3cac6a1, 9fd490db62ed4935a42f8ae3674c6ced; no voters:
I20260926 03:41:02.197418 16842 raft_consensus.cc:2781] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:41:02.197678 16842 raft_consensus.cc:482] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:41:02.197785 16842 raft_consensus.cc:3037] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:02.200562 16842 raft_consensus.cc:504] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
I20260926 03:41:02.201367 16842 leader_election.cc:288] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399), abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169)
I20260926 03:41:02.202099 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca7205460346f389c9835da42aaa86" candidate_uuid: "277941c72bff47e9a2a26623a3cac6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd490db62ed4935a42f8ae3674c6ced"
I20260926 03:41:02.202214 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca7205460346f389c9835da42aaa86" candidate_uuid: "277941c72bff47e9a2a26623a3cac6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abccda8aa8d64241999e46b38546dcab"
I20260926 03:41:02.202343 16642 raft_consensus.cc:3037] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:02.202493 16773 raft_consensus.cc:3037] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:02.204865 16642 raft_consensus.cc:2454] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277941c72bff47e9a2a26623a3cac6a1 in term 1.
I20260926 03:41:02.205291 16773 raft_consensus.cc:2454] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277941c72bff47e9a2a26623a3cac6a1 in term 1.
I20260926 03:41:02.205370 16446 leader_election.cc:302] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277941c72bff47e9a2a26623a3cac6a1, 9fd490db62ed4935a42f8ae3674c6ced; no voters:
I20260926 03:41:02.205826 16842 raft_consensus.cc:2781] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:41:02.206374 16842 raft_consensus.cc:686] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 LEADER]: Becoming Leader. State: Replica: 277941c72bff47e9a2a26623a3cac6a1, State: Running, Role: LEADER
I20260926 03:41:02.206979 16842 consensus_queue.cc:237] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } }
W20260926 03:41:02.211882 16688 tablet.cc:2369] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:41:02.213495 16369 catalog_manager.cc:5495] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "277941c72bff47e9a2a26623a3cac6a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "277941c72bff47e9a2a26623a3cac6a1" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43387 } health_report { overall_health: HEALTHY } } }
W20260926 03:41:02.224911 16819 tablet.cc:2369] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:41:02.226382 16556 tablet.cc:2369] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:41:02.274008 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.266060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.1.221.129:43387)
0926 03:41:02.266386 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.266392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.266418 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:41:02.266623 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.266628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.266736 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.266925 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:41:02.267538 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.268503 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.270340 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.270937 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.271093 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.272241 (+ 1148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:41:02.273004 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:41:02.273177 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.273473 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.273601 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 03:41:02.284801 16642 raft_consensus.cc:1261] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Refusing update from remote peer 277941c72bff47e9a2a26623a3cac6a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:02.284909 16773 raft_consensus.cc:1261] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Refusing update from remote peer 277941c72bff47e9a2a26623a3cac6a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:02.285998 16845 consensus_queue.cc:1035] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fd490db62ed4935a42f8ae3674c6ced" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:41:02.286489 16842 consensus_queue.cc:1035] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abccda8aa8d64241999e46b38546dcab" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:41:02.324815 16369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:02.325078 16369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:02.329504 16369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Sending DeleteTablet for 3 replicas of tablet 01ca7205460346f389c9835da42aaa86
I20260926 03:41:02.331076 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16407
I20260926 03:41:02.330998 16622 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca7205460346f389c9835da42aaa86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:02 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260926 03:41:02.331350 16753 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca7205460346f389c9835da42aaa86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:02 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57890
I20260926 03:41:02.332140 16490 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca7205460346f389c9835da42aaa86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:02 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38318
I20260926 03:41:02.344300 16858 tablet_replica.cc:330] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: stopping tablet replica
I20260926 03:41:02.345019 16859 tablet_replica.cc:330] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: stopping tablet replica
I20260926 03:41:02.345537 16859 raft_consensus.cc:2229] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:02.346274 16860 tablet_replica.cc:330] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: stopping tablet replica
I20260926 03:41:02.346863 16860 raft_consensus.cc:2229] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:41:02.347833 16860 raft_consensus.cc:2258] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:02.348029 16859 raft_consensus.cc:2258] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:02.346984 16858 raft_consensus.cc:2229] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:02.349543 16858 raft_consensus.cc:2258] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:02.350598 16860 ts_tablet_manager.cc:1905] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:02.351454 16859 ts_tablet_manager.cc:1905] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:02.355832 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.333001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.1.221.129:43387)
0926 03:41:02.333208 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.333214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.333236 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:41:02.333316 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.333322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.333414 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.333558 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:02.335744 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.337364 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.339648 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.340497 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.340656 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.344804 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:02.344825 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:02.344853 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:02.344903 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:02.349929 (+ 5026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.351519 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.351532 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.351535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.351782 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.352449 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.352455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.352458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.353094 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:02.353507 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.355014 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.355372 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:41:02.356890 16550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:02.358227 16858 ts_tablet_manager.cc:1905] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:02.361665 16859 ts_tablet_manager.cc:1918] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:02.361889 16859 log.cc:1196] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/01ca7205460346f389c9835da42aaa86
I20260926 03:41:02.362551 16859 ts_tablet_manager.cc:1939] T 01ca7205460346f389c9835da42aaa86 P abccda8aa8d64241999e46b38546dcab: Deleting consensus metadata
I20260926 03:41:02.365231 16357 catalog_manager.cc:4841] TS abccda8aa8d64241999e46b38546dcab (127.1.221.131:45169): tablet 01ca7205460346f389c9835da42aaa86 (table test-table [id=e4751132b34440adaf7759b80fddc2c3]) successfully deleted
I20260926 03:41:02.365823 16858 ts_tablet_manager.cc:1918] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:02.365986 16858 log.cc:1196] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/01ca7205460346f389c9835da42aaa86
I20260926 03:41:02.366410 16858 ts_tablet_manager.cc:1939] T 01ca7205460346f389c9835da42aaa86 P 9fd490db62ed4935a42f8ae3674c6ced: Deleting consensus metadata
I20260926 03:41:02.368295 16357 catalog_manager.cc:4841] TS 9fd490db62ed4935a42f8ae3674c6ced (127.1.221.130:46399): tablet 01ca7205460346f389c9835da42aaa86 (table test-table [id=e4751132b34440adaf7759b80fddc2c3]) successfully deleted
I20260926 03:41:02.464064 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16407
W20260926 03:41:02.481292 16357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:43387 (error 108)
I20260926 03:41:02.481541 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16559
I20260926 03:41:02.483433 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.482844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38360)
0926 03:41:02.483033 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.483197 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:41:02.483701 16357 catalog_manager.cc:4584] TS 277941c72bff47e9a2a26623a3cac6a1 (127.1.221.129:43387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01ca7205460346f389c9835da42aaa86: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
I20260926 03:41:02.498476 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.482517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.1.221.130:46399)
0926 03:41:02.482707 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.482714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.482740 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:41:02.482883 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.482889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.482995 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.483107 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:02.484827 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.485631 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.487093 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.487544 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.487661 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.490509 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:02.490524 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:02.490541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:02.490576 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:02.494337 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.495324 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.495333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.495336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.495690 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.496243 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.496248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.496251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.496864 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:02.497150 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.497540 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.497922 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:41:02.499585 16682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:02.537782 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.536947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38370)
0926 03:41:02.537316 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.537516 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 03:41:02.539325 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.538783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38380)
0926 03:41:02.538990 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.539135 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:41:02.600715 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16559
I20260926 03:41:02.601056 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.600476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38390)
0926 03:41:02.600701 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.600831 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:41:02.603358 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.602633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38406)
0926 03:41:02.602975 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.603112 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 03:41:02.620035 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16690
I20260926 03:41:02.642170 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.625646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.1.221.131:45169)
0926 03:41:02.625865 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:02.625870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:02.625893 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:41:02.625977 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:02.625981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:02.626066 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:02.626204 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:02.627860 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.628817 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.630401 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:02.630989 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:02.631158 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:02.634624 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:02.634652 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:02.634675 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:02.634751 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:02.638742 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.639833 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.639842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.639844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.640021 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:02.640453 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:02.640457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:02.640459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:02.640854 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:02.641170 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:02.641524 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:41:02.641776 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:41:02.643138 16813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:02.685002 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.684103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38412)
0926 03:41:02.684496 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.684678 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 03:41:02.686775 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.686141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38426)
0926 03:41:02.686423 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.686563 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 03:41:02.750573 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16690
I20260926 03:41:02.767333 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16337
I20260926 03:41:02.769089 16400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:02.899750 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16337
I20260926 03:41:02.899941 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:02.899117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43387 (local address 127.0.0.1:38434)
0926 03:41:02.899543 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:02.899681 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:41:02.928223 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--webserver_interface=127.1.221.190
--webserver_port=36081
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:03.309959 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:03.310292 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:03.310384 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:03.318024 16866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:41:03.318156 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:03.318203 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:03.318267 16866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:41:03.318336 16866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:41:03.330513 16866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=36081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:03.332954 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:03.334501 16866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:03.349364 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:03.349414 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:03.349364 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:03.361876 16866 server_base.cc:1034] running on GCE node
I20260926 03:41:03.362895 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:03.364686 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:03.366112 16866 hybrid_clock.cc:648] HybridClock initialized: now 1790394063366038 us; error 64 us; skew 500 ppm
I20260926 03:41:03.366564 16866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:03 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:03.370496 16866 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:41:03.372575 16866 webserver.cc:460] Webserver started at http://127.1.221.190:36081/ using document root <none> and password file <none>
I20260926 03:41:03.373385 16866 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:03.373523 16866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:03.380229 16866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 03:41:03.384155 16881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:03.386365 16866 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:41:03.386618 16866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "5162b4a1bb5248aaa9d5615d962715e6"
format_stamp: "Formatted at 2026-09-26 03:40:56 on dist-test-slave-lqrk"
server_key: "b3a9423ae6ebcfbc0abbaa93307cb1ee"
server_key_iv: "fee04a67e9699e645471cacbd201e67d"
server_key_version: "encryptionkey@0"
I20260926 03:41:03.387791 16866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:03.405642 16866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:03.410733 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:03.411044 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:03.431749 16866 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34117
I20260926 03:41:03.431830 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34117 every 8 connection(s)
I20260926 03:41:03.433425 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:41:03.442432 16933 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:41:03.443257 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16866
I20260926 03:41:03.444167 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:43387
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=32929
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:41:03.446722 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap starting.
I20260926 03:41:03.472517 16933 log.cc:824] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:03.492460 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:41:03.493166 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6: Bootstrap complete.
I20260926 03:41:03.503996 16933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:03.505155 16933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Initialized, Role: FOLLOWER
I20260926 03:41:03.505970 16933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:03.506253 16933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:41:03.506378 16933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:41:03.506564 16933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:41:03.511355 16933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:03.512117 16933 leader_election.cc:302] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5162b4a1bb5248aaa9d5615d962715e6; no voters:
I20260926 03:41:03.512915 16933 leader_election.cc:288] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:41:03.513094 16937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:41:03.514015 16937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [term 3 LEADER]: Becoming Leader. State: Replica: 5162b4a1bb5248aaa9d5615d962715e6, State: Running, Role: LEADER
I20260926 03:41:03.514974 16937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } }
I20260926 03:41:03.515678 16933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:41:03.518211 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:41:03.518285 16938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5162b4a1bb5248aaa9d5615d962715e6. Latest consensus state: current_term: 3 leader_uuid: "5162b4a1bb5248aaa9d5615d962715e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5162b4a1bb5248aaa9d5615d962715e6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34117 } } }
I20260926 03:41:03.518826 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:03.518823 16938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5162b4a1bb5248aaa9d5615d962715e6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:03.520046 16942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:41:03.537793 16942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:41:03.539602 16942 catalog_manager.cc:1259] Loaded cluster ID: 33460f8e96344dd6bea74971afd8857f
I20260926 03:41:03.539834 16942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 03:41:03.542402 16942 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb993ec99ad google::LogMessage::Fail() at ??:0
@ 0x7fb993ecc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb993ec94de google::LogMessage::Flush() at ??:0
@ 0x7fb993ecaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9a98fc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb9a98f9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9a9981610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb9a9981439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb996b15345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb996b1891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb996b18799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb996af3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb999dcd6db start_thread at ??:0
@ 0x7fb992d6361f clone at ??:0
W20260926 03:41:03.851037 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:03.851464 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:03.851643 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:03.860140 16935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:03.860284 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:03.860438 16935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:41:03.871577 16935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:43387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=32929
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:03.872987 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:03.874423 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:03.889191 16975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:03.889214 16972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:03.889190 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:03.889997 16935 server_base.cc:1034] running on GCE node
I20260926 03:41:03.890738 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:03.892218 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:03.893512 16935 hybrid_clock.cc:648] HybridClock initialized: now 1790394063893433 us; error 62 us; skew 500 ppm
I20260926 03:41:03.893882 16935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:03 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:03.897528 16935 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:41:03.899310 16935 webserver.cc:460] Webserver started at http://127.1.221.129:32929/ using document root <none> and password file <none>
I20260926 03:41:03.900105 16935 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:03.900231 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:03.906598 16935 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 03:41:03.910395 16982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:03.912400 16935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:41:03.912611 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "277941c72bff47e9a2a26623a3cac6a1"
format_stamp: "Formatted at 2026-09-26 03:40:57 on dist-test-slave-lqrk"
server_key: "15b9182a15349f3772c1ec2dabc5de98"
server_key_iv: "e0a88f8eb4b139531d732e2a4c74f511"
server_key_version: "encryptionkey@0"
I20260926 03:41:03.913722 16935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:03.929996 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:03.935020 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:03.935328 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:03.936456 16935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:03.938515 16989 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 03:41:03.944650 16989 ts_tablet_manager.cc:1852] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 03:41:03.945973 16989 ts_tablet_manager.cc:1905] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:03.954677 16989 ts_tablet_manager.cc:1918] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:03.954936 16989 log.cc:1196] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/01ca7205460346f389c9835da42aaa86
I20260926 03:41:03.955549 16989 ts_tablet_manager.cc:1939] T 01ca7205460346f389c9835da42aaa86 P 277941c72bff47e9a2a26623a3cac6a1: Deleting consensus metadata
I20260926 03:41:03.956008 16935 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 03:41:03.956137 16935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260926 03:41:03.956286 16935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:03.956382 16935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:03.991871 16935 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43387
I20260926 03:41:03.991989 17095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43387 every 8 connection(s)
I20260926 03:41:03.993266 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:41:03.994709 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 16935
I20260926 03:41:03.995632 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:46399
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=33281
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:41:03.999611 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:03.997269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:58163)
0926 03:41:03.998255 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:03.998517 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":394,"threads_started":1}
W20260926 03:41:04.000509 17096 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:34117 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:34117: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
I20260926 03:41:04.001704 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:04.001278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:48375)
0926 03:41:04.001492 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:04.001574 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:41:04.002851 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:04.002400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:60007)
0926 03:41:04.002608 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:04.002705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 03:41:04.003284 17096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:41:04.346470 17099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:04.346800 17099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:04.346935 17099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:04.354676 17099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:04.354817 17099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:04.354984 17099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:41:04.368381 17099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:46399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=33281
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:04.370131 17099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:04.371726 17099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:04.386013 17107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:04.386025 17104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:04.386027 17105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:04.387234 17099 server_base.cc:1034] running on GCE node
I20260926 03:41:04.387988 17099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:04.389034 17099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:04.390257 17099 hybrid_clock.cc:648] HybridClock initialized: now 1790394064390210 us; error 37 us; skew 500 ppm
I20260926 03:41:04.390609 17099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:04 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:04.393951 17099 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:41:04.395687 17099 webserver.cc:460] Webserver started at http://127.1.221.130:33281/ using document root <none> and password file <none>
I20260926 03:41:04.396430 17099 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:04.396554 17099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:04.402474 17099 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 03:41:04.405853 17114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:04.407581 17099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:41:04.407780 17099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "9fd490db62ed4935a42f8ae3674c6ced"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "74a93cd8f4ede0d6a569c821061efa26"
server_key_iv: "a6480759008a6dbd1d880654f17ddace"
server_key_version: "encryptionkey@0"
I20260926 03:41:04.409089 17099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:04.425988 17099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:04.431667 17099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:04.431981 17099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:04.433095 17099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:04.434726 17099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:04.434842 17099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:04.434962 17099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:04.435038 17099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:04.471289 17099 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:46399
I20260926 03:41:04.471349 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:46399 every 8 connection(s)
I20260926 03:41:04.473541 17099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:41:04.478955 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:04.476963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:44287)
0926 03:41:04.477785 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:04.478046 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
W20260926 03:41:04.479818 17227 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:34117 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:34117: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
I20260926 03:41:04.480829 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:04.480453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:60579)
0926 03:41:04.480622 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:04.480709 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:41:04.481740 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:04.481399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:45051)
0926 03:41:04.481531 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:04.481627 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 03:41:04.481977 17227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:41:04.482033 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17099
I20260926 03:41:04.482645 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:45169
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:34117
--webserver_port=35327
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:04.857810 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:04.858140 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:04.858246 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:04.866348 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:04.866541 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:04.866708 17230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:41:04.877612 17230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:45169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=35327
--tserver_master_addrs=127.1.221.190:34117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:04.879022 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:04.880461 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:04.900173 17235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:04.900315 17236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:04.900833 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:04.901634 17230 server_base.cc:1034] running on GCE node
I20260926 03:41:04.902554 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:04.903868 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:04.905730 17230 hybrid_clock.cc:648] HybridClock initialized: now 1790394064905448 us; error 75 us; skew 500 ppm
I20260926 03:41:04.906209 17230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:04 dist-test-slave-lqrk krb5kdc[15785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:04.917371 17230 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:41:04.919795 17230 webserver.cc:460] Webserver started at http://127.1.221.131:35327/ using document root <none> and password file <none>
I20260926 03:41:04.920612 17230 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:04.920751 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:04.928746 17230 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 03:41:04.932741 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:04.934794 17230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:41:04.935048 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "abccda8aa8d64241999e46b38546dcab"
format_stamp: "Formatted at 2026-09-26 03:40:58 on dist-test-slave-lqrk"
server_key: "c2fbeebff726e9ffbec9a5b2ebf5999e"
server_key_iv: "6a2938149156fbfb31417e4370b0a647"
server_key_version: "encryptionkey@0"
I20260926 03:41:04.936332 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:04.962147 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:04.969656 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:04.969951 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:04.971170 17230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:04.972851 17230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:04.972992 17230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:04.973117 17230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:04.973196 17230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:05.006059 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:05.004339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:41281)
0926 03:41:05.004941 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:05.005162 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":301,"threads_started":1}
I20260926 03:41:05.017901 17230 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:45169
I20260926 03:41:05.018071 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:45169 every 8 connection(s)
I20260926 03:41:05.019831 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:41:05.020665 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17230
I20260926 03:41:05.027943 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:05.025120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:33931)
0926 03:41:05.026388 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:05.026745 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":479,"threads_started":1}
W20260926 03:41:05.028970 17359 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:34117 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:34117: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
I20260926 03:41:05.030310 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:05.029807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:49267)
0926 03:41:05.030076 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:05.030177 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 03:41:05.031543 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:05.031032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:35167)
0926 03:41:05.031228 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:05.031321 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 03:41:05.031838 17359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:41:05.484323 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:05.482962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:32973)
0926 03:41:05.483799 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:05.483982 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":401,"threads_started":1}
I20260926 03:41:06.009356 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:06.007935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:41625)
0926 03:41:06.008786 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:06.008978 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":415,"threads_started":1}
I20260926 03:41:06.034355 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:06.032943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:59663)
0926 03:41:06.033678 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:06.033909 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":359,"threads_started":1}
I20260926 03:41:06.487538 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:06.486231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:40401)
0926 03:41:06.487088 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:06.487237 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":447,"threads_started":1}
I20260926 03:41:07.012207 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:07.011166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:50255)
0926 03:41:07.011828 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:07.011961 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":355,"threads_started":1}
I20260926 03:41:07.037613 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:07.036286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:57353)
0926 03:41:07.037226 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:07.037353 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":451,"threads_started":1}
I20260926 03:41:07.490377 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:07.489143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:37797)
0926 03:41:07.489934 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:07.490140 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":346,"threads_started":1}
I20260926 03:41:08.014906 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:08.013779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:36055)
0926 03:41:08.014530 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:08.014666 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":427,"threads_started":1}
I20260926 03:41:08.041019 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:08.039432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:47989)
0926 03:41:08.040471 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:08.040640 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":494,"threads_started":1}
I20260926 03:41:08.493083 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:08.491955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:38911)
0926 03:41:08.492678 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:08.492802 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":368,"threads_started":1}
I20260926 03:41:09.017907 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:09.016502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:54873)
0926 03:41:09.017371 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:09.017582 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":376,"threads_started":1}
I20260926 03:41:09.044167 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:09.042892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:35631)
0926 03:41:09.043774 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:09.043916 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":390,"threads_started":1}
I20260926 03:41:09.495868 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:09.494643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:33311)
0926 03:41:09.495463 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:09.495610 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":397,"threads_started":1}
I20260926 03:41:10.020710 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:10.019570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:39591)
0926 03:41:10.020274 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:10.020431 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260926 03:41:10.046737 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:10.045665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:60117)
0926 03:41:10.046385 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:10.046520 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":378,"threads_started":1}
I20260926 03:41:10.498472 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:10.497401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:57149)
0926 03:41:10.498075 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:10.498199 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":344,"threads_started":1}
I20260926 03:41:11.023669 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:11.022272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:34859)
0926 03:41:11.023142 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:11.023283 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":350,"threads_started":1}
I20260926 03:41:11.049124 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:11.048287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:59637)
0926 03:41:11.048824 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:11.048930 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":286,"threads_started":1}
I20260926 03:41:11.501369 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:11.500069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:60139)
0926 03:41:11.500838 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:11.500982 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":354,"threads_started":1}
I20260926 03:41:12.026589 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:12.025248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:43779)
0926 03:41:12.025938 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:12.026155 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":337,"threads_started":1}
I20260926 03:41:12.051697 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:12.050411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:58445)
0926 03:41:12.051290 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:12.051451 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":384,"threads_started":1}
I20260926 03:41:12.504163 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:12.502965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:51747)
0926 03:41:12.503759 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:12.503889 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":441,"threads_started":1}
I20260926 03:41:13.028964 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:13.027839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:37211)
0926 03:41:13.028548 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:13.028714 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":324,"threads_started":1}
I20260926 03:41:13.054153 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:13.053199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:51499)
0926 03:41:13.053852 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:13.053971 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":334,"threads_started":1}
I20260926 03:41:13.506709 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:13.505626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:56015)
0926 03:41:13.506388 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:13.506506 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":333,"threads_started":1}
I20260926 03:41:14.031674 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:14.030530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:55433)
0926 03:41:14.031266 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:14.031454 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":360,"threads_started":1}
I20260926 03:41:14.056891 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:14.055787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:59941)
0926 03:41:14.056521 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:14.056655 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260926 03:41:14.509030 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:14.508124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:48415)
0926 03:41:14.508748 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:14.508848 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":332,"threads_started":1}
I20260926 03:41:15.034567 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:15.033221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:59877)
0926 03:41:15.033921 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:15.034221 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":383,"threads_started":1}
I20260926 03:41:15.059552 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:15.058451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:57051)
0926 03:41:15.059161 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:15.059275 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":377,"threads_started":1}
I20260926 03:41:15.511274 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:15.510109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:43253)
0926 03:41:15.510820 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:15.510973 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
I20260926 03:41:16.037235 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:16.036178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:40975)
0926 03:41:16.036868 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:16.037006 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":361,"threads_started":1}
I20260926 03:41:16.062214 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:16.060972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:34145)
0926 03:41:16.061744 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:16.061888 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":394,"threads_started":1}
I20260926 03:41:16.513728 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:16.512717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:53795)
0926 03:41:16.513372 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:16.513498 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260926 03:41:17.040105 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:17.038742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:47661)
0926 03:41:17.039649 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:17.039836 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":357,"threads_started":1}
I20260926 03:41:17.064455 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:17.063576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:50023)
0926 03:41:17.064107 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:17.064213 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":290,"threads_started":1}
I20260926 03:41:17.516144 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:17.515059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:36677)
0926 03:41:17.515787 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:17.515921 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":395,"threads_started":1}
I20260926 03:41:18.042495 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:18.041391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:51431)
0926 03:41:18.042073 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:18.042221 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":340,"threads_started":1}
I20260926 03:41:18.066790 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:18.065818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:52199)
0926 03:41:18.066472 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:18.066594 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
I20260926 03:41:18.518385 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:18.517427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:33105)
0926 03:41:18.517980 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:18.518109 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
I20260926 03:41:19.045459 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:19.044221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:51961)
0926 03:41:19.045040 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:19.045189 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":374,"threads_started":1}
I20260926 03:41:19.069231 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:19.068041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:54939)
0926 03:41:19.068787 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:19.068929 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":374,"threads_started":1}
I20260926 03:41:19.521178 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:19.520115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.130:56923)
0926 03:41:19.520785 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:19.520925 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":332,"threads_started":1}
I20260926 03:41:20.028784 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 16935
I20260926 03:41:20.048401 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:20.047002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.129:48491)
0926 03:41:20.047848 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:20.048069 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":396,"threads_started":1}
I20260926 03:41:20.060493 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:20.030236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.1.221.129:43387)
0926 03:41:20.031119 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:20.031143 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:20.031210 (+ 67us) server_negotiation.cc:407] Connection header received
0926 03:41:20.033589 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:20.033604 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:20.034220 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:20.034656 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:20.037103 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.038821 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.040997 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.042119 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.042346 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:20.052534 (+ 10188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:20.052547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:20.052569 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:20.052639 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:20.055989 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.057187 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.057196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.057204 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.057450 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.058007 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.058012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.058015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.058700 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:20.059020 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:20.059596 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 03:41:20.060102 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":424,"threads_started":1}
I20260926 03:41:20.062255 17091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:20.072009 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:20.070930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34117 (local address 127.1.221.131:54605)
0926 03:41:20.071607 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:20.071774 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":378,"threads_started":1}
I20260926 03:41:20.155073 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 16935
I20260926 03:41:20.171047 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17099
I20260926 03:41:20.194149 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:20.172285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.1.221.130:46399)
0926 03:41:20.172968 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:20.172987 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:20.173038 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:41:20.174905 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:20.174919 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:20.175624 (+ 705us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:20.176073 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:20.177932 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.179456 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.180963 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.181892 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.182121 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:20.185456 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:20.185469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:20.185496 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:20.185569 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:20.189525 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.190603 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.190613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.190622 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.190871 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.191347 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.191353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.191357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.192210 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:20.192608 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:20.193150 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 03:41:20.193776 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260926 03:41:20.195945 17222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:20.286522 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17099
I20260926 03:41:20.306303 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17230
I20260926 03:41:20.334537 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:20.307789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.1.221.131:45169)
0926 03:41:20.308721 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:20.308741 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:20.308802 (+ 61us) server_negotiation.cc:407] Connection header received
0926 03:41:20.310955 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:20.310972 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:20.311780 (+ 808us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:20.312254 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:20.314962 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.316870 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.318705 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:20.319968 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:20.320279 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:20.324287 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:20.324301 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:20.324341 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:20.324418 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:20.329080 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.330600 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.330615 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.330631 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.330954 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:20.331617 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:20.331622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:20.331625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:20.332514 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:20.332891 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:20.333519 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 03:41:20.334124 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":481,"threads_started":1}
I20260926 03:41:20.336778 17354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:20.427214 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17230
2026-09-26T03:41:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25540 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:41:20 dist-test-slave-lqrk krb5kdc[17416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:41:20 dist-test-slave-lqrk krb5kdc[17416](info): set up 2 sockets
Sep 26 03:41:20 dist-test-slave-lqrk krb5kdc[17416](info): commencing operation
krb5kdc: starting...
W20260926 03:41:21.500501 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:41:21 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:41:21.801393 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:22.182835 17429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:22.183183 17429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:22.183281 17429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:22.191033 17429 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:41:22.191150 17429 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:41:22.191188 17429 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:41:22.191232 17429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:41:22.191282 17429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:22.191371 17429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:22.191478 17429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:41:22.191535 17429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:41:22.202062 17429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:22.203495 17429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:22.204874 17429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:22.219803 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:22.219858 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:22.219756 17434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:22.220643 17429 server_base.cc:1034] running on GCE node
I20260926 03:41:22.221675 17429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:22.223346 17429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:22.224740 17429 hybrid_clock.cc:648] HybridClock initialized: now 1790394082224639 us; error 60 us; skew 500 ppm
I20260926 03:41:22.225162 17429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:22 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:22.228641 17429 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:41:22.230516 17429 webserver.cc:460] Webserver started at http://127.1.221.190:36575/ using document root <none> and password file <none>
I20260926 03:41:22.231273 17429 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:22.231411 17429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:22.231746 17429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:41:22.234905 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "089ce4b5c5cf472087dc53a02679ac8d"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "1289f4f3ea071f7c11811d2730d0277e"
server_key_iv: "3a00e8faf8ae183503fad9b13c0c5bf3"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.235764 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "089ce4b5c5cf472087dc53a02679ac8d"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "1289f4f3ea071f7c11811d2730d0277e"
server_key_iv: "3a00e8faf8ae183503fad9b13c0c5bf3"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.242054 17429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:41:22.247473 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:22.249917 17429 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:41:22.250160 17429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "089ce4b5c5cf472087dc53a02679ac8d"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "1289f4f3ea071f7c11811d2730d0277e"
server_key_iv: "3a00e8faf8ae183503fad9b13c0c5bf3"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.250401 17429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:22.264940 17429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:22.269673 17429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:22.269929 17429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:22.291581 17429 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40107
I20260926 03:41:22.291610 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40107 every 8 connection(s)
I20260926 03:41:22.293243 17429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:41:22.298666 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:22.303303 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17429
I20260926 03:41:22.303622 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:41:22.304515 1910 external_mini_cluster.cc:1428] Setting key 38a3ded9c02d35563bab370d1afa0d54
I20260926 03:41:22.315528 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap starting.
I20260926 03:41:22.319967 17496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Neither blocks nor log segments found. Creating new log.
Sep 26 03:41:22 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:22.321383 17496 log.cc:824] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:22.325352 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: No bootstrap required, opened a new log
I20260926 03:41:22.333288 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:22.307537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.1.221.190:40107)
0926 03:41:22.308643 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:22.308660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:22.308719 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:41:22.310776 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:22.310810 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:22.311425 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:22.311935 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:22.313633 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.315117 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.316705 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.317554 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.317768 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:22.322220 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:22.322234 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:22.322269 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:22.322355 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:22.326866 (+ 4511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:22.328095 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:22.328105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:22.328115 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:22.328491 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:22.329066 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:22.329071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:22.329074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:22.330033 (+ 959us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:22.330383 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:22.330973 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 03:41:22.331721 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":476,"threads_started":1}
I20260926 03:41:22.335470 17496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:22.336067 17496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:22.336187 17496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Initialized, Role: FOLLOWER
I20260926 03:41:22.336962 17496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:22.337265 17496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:41:22.337435 17496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:41:22.337626 17496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:22.340157 17496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:22.340727 17496 leader_election.cc:302] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089ce4b5c5cf472087dc53a02679ac8d; no voters:
I20260926 03:41:22.341540 17496 leader_election.cc:288] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:41:22.341883 17501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:41:22.343056 17501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 LEADER]: Becoming Leader. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Running, Role: LEADER
I20260926 03:41:22.343837 17501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:22.344100 17496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:41:22.347451 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089ce4b5c5cf472087dc53a02679ac8d. Latest consensus state: current_term: 1 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:41:22.347945 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:22.348582 17502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:41:22.348944 17502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:22.349778 17509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:41:22.357259 17509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:41:22.369211 17509 catalog_manager.cc:1347] Generated new cluster ID: 957677ab246b4fe1b55d0869d27537ad
I20260926 03:41:22.369364 17509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:41:22.393371 17509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:41:22.395179 17509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:41:22.402765 17509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Generated new TSK 0
I20260926 03:41:22.403779 17509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:41:22.472043 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:22.824065 17522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:22.824429 17522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:22.824558 17522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:22.833141 17522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:22.833277 17522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:22.833421 17522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:41:22.843961 17522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:22.845432 17522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:22.846772 17522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:22.862838 17527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:22.862761 17530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:22.862722 17528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:22.863886 17522 server_base.cc:1034] running on GCE node
I20260926 03:41:22.864630 17522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:22.865805 17522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:22.867072 17522 hybrid_clock.cc:648] HybridClock initialized: now 1790394082866996 us; error 64 us; skew 500 ppm
I20260926 03:41:22.867527 17522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:22 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:22.871145 17522 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:41:22.873028 17522 webserver.cc:460] Webserver started at http://127.1.221.129:45339/ using document root <none> and password file <none>
I20260926 03:41:22.873749 17522 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:22.873862 17522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:22.874213 17522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:41:22.877314 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "9e9cf2b183274494bb665c023aeb4dae"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "d04e4befcba325ccec4359d04788e22c"
server_key_iv: "93acaa6ebe4fd0be077c0ba7d4e18d97"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.878127 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "9e9cf2b183274494bb665c023aeb4dae"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "d04e4befcba325ccec4359d04788e22c"
server_key_iv: "93acaa6ebe4fd0be077c0ba7d4e18d97"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.884200 17522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:41:22.888796 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:22.891052 17522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 03:41:22.891359 17522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "9e9cf2b183274494bb665c023aeb4dae"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "d04e4befcba325ccec4359d04788e22c"
server_key_iv: "93acaa6ebe4fd0be077c0ba7d4e18d97"
server_key_version: "encryptionkey@0"
I20260926 03:41:22.891656 17522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:22.908895 17522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:22.914445 17522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:22.914804 17522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:22.916189 17522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:22.918073 17522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:22.918175 17522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:22.918305 17522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:22.918375 17522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:22.958487 17522 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41763
I20260926 03:41:22.958514 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41763 every 8 connection(s)
I20260926 03:41:22.960275 17522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:41:22.967351 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17522
I20260926 03:41:22.967633 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:41:22.968317 1910 external_mini_cluster.cc:1428] Setting key fa6461c5e1890fe6c66973fa6da2c806
Sep 26 03:41:22 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:22.987877 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:22.963550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:51273 (local address 127.1.221.190:40107)
0926 03:41:22.964126 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:22.964133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:22.965188 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 03:41:22.967658 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:22.967665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:22.967777 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:22.967893 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:22.970657 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.971820 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.973985 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.974594 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.974763 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:22.979984 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:22.979994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:22.979999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:22.980071 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:22.983724 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:22.984854 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:22.984859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:22.984863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:22.985076 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:22.985616 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:22.985620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:22.985623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:22.986042 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:22.986210 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:22.987099 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 03:41:22.987468 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
I20260926 03:41:22.988667 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:22.963950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:51273)
0926 03:41:22.964941 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:22.965003 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:22.967160 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:22.968282 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:22.968294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:22.969224 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:22.970315 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:22.970351 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.972099 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.972104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:22.973752 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:22.973774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:22.974917 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:22.974923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:22.975236 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:22.976282 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:22.976318 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:22.979555 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:22.984036 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:22.984047 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:22.984070 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:22.984685 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:22.985221 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:22.985224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:22.985227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:22.985493 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:22.986200 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:22.986207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:22.986663 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 03:41:22.986922 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 03:41:22.987458 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":433,"threads_started":1}
I20260926 03:41:22.990530 17650 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:22.990835 17650 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:22.991668 17650 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
I20260926 03:41:22.994156 17461 ts_manager.cc:194] Registered new tserver with Master: 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763)
I20260926 03:41:22.995786 17461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:51273
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:41:23.036038 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:23.386240 17656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:23.386582 17656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:23.386723 17656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:23.394583 17656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:23.394733 17656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:23.394886 17656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:41:23.405938 17656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:23.407320 17656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:23.408731 17656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:23.424145 17664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:23.424110 17661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:23.424592 17662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:23.425045 17656 server_base.cc:1034] running on GCE node
I20260926 03:41:23.425894 17656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:23.427162 17656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:23.428423 17656 hybrid_clock.cc:648] HybridClock initialized: now 1790394083428378 us; error 76 us; skew 500 ppm
I20260926 03:41:23.428851 17656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:23 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:23.432925 17656 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:41:23.435055 17656 webserver.cc:460] Webserver started at http://127.1.221.130:42791/ using document root <none> and password file <none>
I20260926 03:41:23.436086 17656 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:23.436244 17656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:23.436645 17656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:41:23.440742 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "4d23671137764c5c8b9d6e288d4d50bf"
format_stamp: "Formatted at 2026-09-26 03:41:23 on dist-test-slave-lqrk"
server_key: "e71dc4d22ab27ee39f404079899b52b7"
server_key_iv: "0b1702e395c7f91afd392e82a0dbd914"
server_key_version: "encryptionkey@0"
I20260926 03:41:23.441807 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "4d23671137764c5c8b9d6e288d4d50bf"
format_stamp: "Formatted at 2026-09-26 03:41:23 on dist-test-slave-lqrk"
server_key: "e71dc4d22ab27ee39f404079899b52b7"
server_key_iv: "0b1702e395c7f91afd392e82a0dbd914"
server_key_version: "encryptionkey@0"
I20260926 03:41:23.449551 17656 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 03:41:23.454684 17671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:23.456859 17656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:41:23.457096 17656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "4d23671137764c5c8b9d6e288d4d50bf"
format_stamp: "Formatted at 2026-09-26 03:41:23 on dist-test-slave-lqrk"
server_key: "e71dc4d22ab27ee39f404079899b52b7"
server_key_iv: "0b1702e395c7f91afd392e82a0dbd914"
server_key_version: "encryptionkey@0"
I20260926 03:41:23.457356 17656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:23.474474 17656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:23.480094 17656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:23.480405 17656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:23.481550 17656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:23.483070 17656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:23.483173 17656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:23.483304 17656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:23.483362 17656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:23.518749 17656 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45101
I20260926 03:41:23.518896 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45101 every 8 connection(s)
I20260926 03:41:23.520164 17656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:41:23.525683 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17656
I20260926 03:41:23.525874 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:41:23.526340 1910 external_mini_cluster.cc:1428] Setting key cd37eef8009854c9b56a6a53a3b1789d
Sep 26 03:41:23 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:23.548758 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:23.523626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36397 (local address 127.1.221.190:40107)
0926 03:41:23.524151 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:23.524157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:23.525302 (+ 1145us) server_negotiation.cc:407] Connection header received
0926 03:41:23.527938 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:23.527944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:23.528033 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:23.528156 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:23.530701 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:23.531593 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:23.533598 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:23.534114 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:23.534254 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:23.540280 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:23.540289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:23.540294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:23.540354 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:23.543593 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:23.545477 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:23.545483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:23.545485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:23.545680 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:23.546392 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:23.546396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:23.546399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:23.546782 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:23.546937 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:23.548218 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0926 03:41:23.548452 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260926 03:41:23.550889 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:23.524138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:36397)
0926 03:41:23.525032 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:23.525090 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:23.527538 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:23.528544 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:23.528560 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:23.529523 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:23.530430 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:23.530459 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:23.531794 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:23.531799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:23.533364 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:23.533385 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:23.534420 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:23.534429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:23.534732 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:23.535774 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:23.535805 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:23.539762 (+ 3957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:23.544077 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:23.544093 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:23.544124 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:23.545178 (+ 1054us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:23.545840 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:23.545846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:23.545849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:23.546229 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:23.546951 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:23.546962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:23.547659 (+ 697us) client_negotiation.cc:769] Sending connection context
0926 03:41:23.548055 (+ 396us) client_negotiation.cc:240] Negotiation successful
0926 03:41:23.548873 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":322,"threads_started":1}
I20260926 03:41:23.552911 17784 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:23.553450 17784 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:23.554757 17784 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:41:23.556885 17461 ts_manager.cc:194] Registered new tserver with Master: 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101)
I20260926 03:41:23.557790 17461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:36397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:41:23.592095 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:41:23.999307 17650 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
W20260926 03:41:24.016287 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:24.016714 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:24.016875 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:24.025888 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:24.026077 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:24.026253 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:41:24.038133 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:24.040083 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:24.041846 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:24.059755 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:24.059664 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:24.060570 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:24.062628 17790 server_base.cc:1034] running on GCE node
I20260926 03:41:24.063738 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:24.065158 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:24.066484 17790 hybrid_clock.cc:648] HybridClock initialized: now 1790394084066417 us; error 54 us; skew 500 ppm
I20260926 03:41:24.067015 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:24 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:24.071461 17790 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:41:24.073611 17790 webserver.cc:460] Webserver started at http://127.1.221.131:39765/ using document root <none> and password file <none>
I20260926 03:41:24.074512 17790 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:24.074640 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:24.074966 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:41:24.078663 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "54ec71fa785b48b69a8ed8285859bb5f"
format_stamp: "Formatted at 2026-09-26 03:41:24 on dist-test-slave-lqrk"
server_key: "2d7f97e0cc8d129b78361aa5bb6d0f0a"
server_key_iv: "588c2477fee0975f37a8804027a92232"
server_key_version: "encryptionkey@0"
I20260926 03:41:24.079615 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "54ec71fa785b48b69a8ed8285859bb5f"
format_stamp: "Formatted at 2026-09-26 03:41:24 on dist-test-slave-lqrk"
server_key: "2d7f97e0cc8d129b78361aa5bb6d0f0a"
server_key_iv: "588c2477fee0975f37a8804027a92232"
server_key_version: "encryptionkey@0"
I20260926 03:41:24.086351 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:41:24.091732 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:24.094058 17790 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260926 03:41:24.094298 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "54ec71fa785b48b69a8ed8285859bb5f"
format_stamp: "Formatted at 2026-09-26 03:41:24 on dist-test-slave-lqrk"
server_key: "2d7f97e0cc8d129b78361aa5bb6d0f0a"
server_key_iv: "588c2477fee0975f37a8804027a92232"
server_key_version: "encryptionkey@0"
I20260926 03:41:24.094738 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:24.112771 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:24.118438 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:24.118788 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:24.120117 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:24.121824 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:24.121909 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:24.122020 17790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:24.122076 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:24.160223 17790 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44329
I20260926 03:41:24.160279 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44329 every 8 connection(s)
I20260926 03:41:24.161870 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:41:24.163158 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17790
I20260926 03:41:24.163416 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:41:24.163950 1910 external_mini_cluster.cc:1428] Setting key 0755bdcae6a738b1521c308f91472520
Sep 26 03:41:24 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:24.188726 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.165427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:41069 (local address 127.1.221.190:40107)
0926 03:41:24.166034 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.166040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.167366 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 03:41:24.169879 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.169885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.169985 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.170112 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:24.172388 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.173277 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.175418 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.175912 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.176052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.180883 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:24.180889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:24.180895 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:24.180948 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:24.183949 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.185137 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.185143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.185145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.185468 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.186207 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.186212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.186215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.186620 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:24.186784 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.188057 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.188350 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
I20260926 03:41:24.190006 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.166075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:41069)
0926 03:41:24.167131 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.167211 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.169508 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.170464 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.170476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.171273 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:24.172133 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.172157 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.173482 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.173486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.175157 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.175175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.176222 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.176279 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.176597 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.177462 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:24.177502 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:24.180572 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:24.184202 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:24.184215 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:24.184240 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:24.184892 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:24.185675 (+ 783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:24.185679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:24.185682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:24.185981 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:24.186832 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:24.186840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:24.187461 (+ 621us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.187813 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.188469 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":375,"threads_started":1}
I20260926 03:41:24.192159 17918 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:24.192613 17918 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:24.193625 17918 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
I20260926 03:41:24.195621 17460 ts_manager.cc:194] Registered new tserver with Master: 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:24.196600 17460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:41069
I20260926 03:41:24.209045 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:41:24.230027 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.214942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.1.221.190:40107)
0926 03:41:24.215214 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.215222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.215684 (+ 462us) server_negotiation.cc:407] Connection header received
0926 03:41:24.215898 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.215902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.215976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.216076 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:24.217707 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.218520 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.220067 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.220478 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.220593 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.223572 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:24.223579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:24.223582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:24.223625 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:24.226731 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.227720 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.227725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.227727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.227887 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.228312 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.228316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.228318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.228716 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:24.229043 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.229390 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.229668 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 03:41:24.235858 17460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:41:24.243186 17460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:41:24.289229 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.261051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.1.221.130:45101)
0926 03:41:24.280670 (+ 19619us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.280677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.280707 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:41:24.280802 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.280808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.281069 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.281265 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:24.281936 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.283303 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.286455 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.287889 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.288052 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.288284 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.288671 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.288861 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19440,"thread_start_us":19339,"threads_started":1}
I20260926 03:41:24.289289 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.260814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:45101 (local address 127.0.0.1:43680)
0926 03:41:24.276625 (+ 15811us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.276690 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.276959 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.281312 (+ 4353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.281319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.281352 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:24.281739 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.281752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.283494 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.283499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.286231 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.286252 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.288139 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.288146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.288370 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.288435 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.288604 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.288864 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15472,"thread_start_us":14590,"threads_started":1}
I20260926 03:41:24.290738 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.261748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44329 (local address 127.0.0.1:50414)
0926 03:41:24.279943 (+ 18195us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.279960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.280063 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.282476 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.282482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.282514 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:24.283032 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.283046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.284783 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.284789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.287116 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.287134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.289777 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.289784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.289975 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.290014 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.290119 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.290274 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18088,"mutex_wait_us":1,"thread_start_us":14990,"threads_started":1}
I20260926 03:41:24.291981 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.262362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.1.221.129:41763)
0926 03:41:24.282032 (+ 19670us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.282042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.282073 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:41:24.282168 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.282175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.282412 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.282616 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:24.283552 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.285271 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.287679 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.289069 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.289219 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.289846 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.291082 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.291329 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19467,"thread_start_us":19301,"threads_started":1}
I20260926 03:41:24.293335 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.263205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.1.221.131:44329)
0926 03:41:24.281604 (+ 18399us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.281615 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.281670 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:41:24.281803 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.281812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.282197 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.282441 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:24.283362 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.284522 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.287362 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.288902 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.289048 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.289305 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.290247 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.290587 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18165,"thread_start_us":17945,"threads_started":1}
I20260926 03:41:24.292635 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.260981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41763 (local address 127.0.0.1:43204)
0926 03:41:24.277083 (+ 16102us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.277101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.277212 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.282801 (+ 5589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.282806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.282842 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:24.283304 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.283314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.285484 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.285490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.287455 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.287471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.290017 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.290024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.290207 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.290248 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.290337 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.290602 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15936,"mutex_wait_us":8,"thread_start_us":19149,"threads_started":1}
I20260926 03:41:24.294868 17719 tablet_service.cc:1467] Processing CreateTablet for tablet 6e402aee165a4408b5f9835b0e352c67 (DEFAULT_TABLE table=test-table [id=62a4c8fdb56b4297a781b7bb3e247660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:24.297168 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e402aee165a4408b5f9835b0e352c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:24.299173 17853 tablet_service.cc:1467] Processing CreateTablet for tablet 6e402aee165a4408b5f9835b0e352c67 (DEFAULT_TABLE table=test-table [id=62a4c8fdb56b4297a781b7bb3e247660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:24.301292 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e402aee165a4408b5f9835b0e352c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:24.302927 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 6e402aee165a4408b5f9835b0e352c67 (DEFAULT_TABLE table=test-table [id=62a4c8fdb56b4297a781b7bb3e247660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:24.306613 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e402aee165a4408b5f9835b0e352c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:24.317175 17937 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Bootstrap starting.
I20260926 03:41:24.320511 17938 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Bootstrap starting.
I20260926 03:41:24.322283 17937 tablet_bootstrap.cc:654] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:24.322515 17939 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Bootstrap starting.
I20260926 03:41:24.323896 17937 log.cc:824] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:24.325834 17938 tablet_bootstrap.cc:654] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:24.327219 17939 tablet_bootstrap.cc:654] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:24.327575 17938 log.cc:824] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:24.327710 17937 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: No bootstrap required, opened a new log
I20260926 03:41:24.328197 17937 ts_tablet_manager.cc:1397] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 03:41:24.328886 17939 log.cc:824] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:24.335639 17938 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: No bootstrap required, opened a new log
I20260926 03:41:24.335639 17939 tablet_bootstrap.cc:492] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: No bootstrap required, opened a new log
I20260926 03:41:24.336107 17939 ts_tablet_manager.cc:1397] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 03:41:24.336114 17938 ts_tablet_manager.cc:1397] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260926 03:41:24.338554 17937 raft_consensus.cc:348] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.339071 17937 raft_consensus.cc:374] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:24.339176 17937 raft_consensus.cc:729] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d23671137764c5c8b9d6e288d4d50bf, State: Initialized, Role: FOLLOWER
I20260926 03:41:24.339926 17937 consensus_queue.cc:260] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.341374 17784 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
I20260926 03:41:24.342216 17937 ts_tablet_manager.cc:1428] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:41:24.348242 17938 raft_consensus.cc:348] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.348935 17938 raft_consensus.cc:374] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:24.349067 17938 raft_consensus.cc:729] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ec71fa785b48b69a8ed8285859bb5f, State: Initialized, Role: FOLLOWER
I20260926 03:41:24.348829 17939 raft_consensus.cc:348] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.349380 17939 raft_consensus.cc:374] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:24.349498 17939 raft_consensus.cc:729] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9cf2b183274494bb665c023aeb4dae, State: Initialized, Role: FOLLOWER
I20260926 03:41:24.349977 17938 consensus_queue.cc:260] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.350384 17939 consensus_queue.cc:260] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.351791 17918 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
I20260926 03:41:24.352691 17938 ts_tablet_manager.cc:1428] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 03:41:24.353787 17939 ts_tablet_manager.cc:1428] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260926 03:41:24.384413 17945 raft_consensus.cc:482] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:41:24.384840 17945 raft_consensus.cc:504] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.386857 17945 leader_election.cc:288] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101), 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:24.395412 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.387497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49293 (local address 127.1.221.131:44329)
0926 03:41:24.387741 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.387749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.388793 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 03:41:24.389017 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.389021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.389094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.389184 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:24.389957 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.390774 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.393258 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.394345 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.394517 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.394599 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.394967 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.395100 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:41:24.395692 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.387497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44329 (local address 127.1.221.129:49293)
0926 03:41:24.388635 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.388656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.388839 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.389214 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.389218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.389244 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:24.389789 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.389799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.390945 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.390949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.392955 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.392970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.394577 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.394583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.394797 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.394828 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.394925 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.395094 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":7,"thread_start_us":764,"threads_started":1}
I20260926 03:41:24.395776 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.387417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:45101 (local address 127.1.221.129:38677)
0926 03:41:24.388413 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:24.388453 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:24.388679 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:24.389155 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:24.389160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:24.389194 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:24.389658 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.389669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.390776 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.390781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.393094 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:24.393110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.394741 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.394749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:24.394945 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.394972 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:41:24.395061 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:41:24.395226 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":73,"thread_start_us":491,"threads_started":1}
I20260926 03:41:24.395937 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.387765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38677 (local address 127.1.221.130:45101)
0926 03:41:24.388058 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.388065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.388603 (+ 538us) server_negotiation.cc:407] Connection header received
0926 03:41:24.388901 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.388907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.389003 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.389112 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:24.389836 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.390627 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.393393 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.394477 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.394656 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.394742 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.395119 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.395267 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 03:41:24.397121 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e402aee165a4408b5f9835b0e352c67" candidate_uuid: "9e9cf2b183274494bb665c023aeb4dae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d23671137764c5c8b9d6e288d4d50bf" is_pre_election: true
I20260926 03:41:24.397150 17873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e402aee165a4408b5f9835b0e352c67" candidate_uuid: "9e9cf2b183274494bb665c023aeb4dae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ec71fa785b48b69a8ed8285859bb5f" is_pre_election: true
I20260926 03:41:24.397873 17739 raft_consensus.cc:2454] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e9cf2b183274494bb665c023aeb4dae in term 0.
I20260926 03:41:24.397873 17873 raft_consensus.cc:2454] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e9cf2b183274494bb665c023aeb4dae in term 0.
I20260926 03:41:24.398820 17541 leader_election.cc:302] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d23671137764c5c8b9d6e288d4d50bf, 9e9cf2b183274494bb665c023aeb4dae; no voters:
I20260926 03:41:24.399333 17945 raft_consensus.cc:2781] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:41:24.399669 17945 raft_consensus.cc:482] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:41:24.399796 17945 raft_consensus.cc:3037] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:24.403069 17945 raft_consensus.cc:504] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
I20260926 03:41:24.404023 17945 leader_election.cc:288] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [CANDIDATE]: Term 1 election: Requested vote from peers 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101), 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:24.404763 17873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e402aee165a4408b5f9835b0e352c67" candidate_uuid: "9e9cf2b183274494bb665c023aeb4dae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ec71fa785b48b69a8ed8285859bb5f"
I20260926 03:41:24.405038 17873 raft_consensus.cc:3037] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:24.406220 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e402aee165a4408b5f9835b0e352c67" candidate_uuid: "9e9cf2b183274494bb665c023aeb4dae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d23671137764c5c8b9d6e288d4d50bf"
I20260926 03:41:24.406538 17739 raft_consensus.cc:3037] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:24.408560 17873 raft_consensus.cc:2454] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9cf2b183274494bb665c023aeb4dae in term 1.
I20260926 03:41:24.409204 17538 leader_election.cc:302] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ec71fa785b48b69a8ed8285859bb5f, 9e9cf2b183274494bb665c023aeb4dae; no voters:
I20260926 03:41:24.409675 17945 raft_consensus.cc:2781] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:41:24.410243 17739 raft_consensus.cc:2454] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9cf2b183274494bb665c023aeb4dae in term 1.
I20260926 03:41:24.410429 17945 raft_consensus.cc:686] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 1 LEADER]: Becoming Leader. State: Replica: 9e9cf2b183274494bb665c023aeb4dae, State: Running, Role: LEADER
I20260926 03:41:24.411278 17945 consensus_queue.cc:237] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } }
W20260926 03:41:24.414031 17919 tablet.cc:2369] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:41:24.418720 17461 catalog_manager.cc:5495] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129). New cstate: current_term: 1 leader_uuid: "9e9cf2b183274494bb665c023aeb4dae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } health_report { overall_health: UNKNOWN } } }
I20260926 03:41:24.441397 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.434359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.1.221.129:41763)
0926 03:41:24.434654 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.434662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.434690 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:41:24.434815 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.434821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.434932 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.435115 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:41:24.435855 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.436743 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.438363 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.438922 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.439047 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.439834 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:41:24.440600 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:41:24.440747 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.440937 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.441049 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:41:24.452636 17739 raft_consensus.cc:1261] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Refusing update from remote peer 9e9cf2b183274494bb665c023aeb4dae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:24.452652 17873 raft_consensus.cc:1261] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Refusing update from remote peer 9e9cf2b183274494bb665c023aeb4dae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:24.454195 17948 consensus_queue.cc:1035] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:41:24.454697 17945 consensus_queue.cc:1035] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:41:24.463490 17651 tablet.cc:2369] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:41:24.492200 17460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:24.492530 17460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:24.497311 17460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Sending DeleteTablet for 3 replicas of tablet 6e402aee165a4408b5f9835b0e352c67
I20260926 03:41:24.499058 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e402aee165a4408b5f9835b0e352c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:24 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43204
I20260926 03:41:24.499248 17719 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e402aee165a4408b5f9835b0e352c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:24 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43680
I20260926 03:41:24.499954 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17522
I20260926 03:41:24.499984 17853 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e402aee165a4408b5f9835b0e352c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:24 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:50414
I20260926 03:41:24.509819 17961 tablet_replica.cc:330] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: stopping tablet replica
I20260926 03:41:24.510406 17961 raft_consensus.cc:2229] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:24.510993 17961 raft_consensus.cc:2258] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:24.511337 17962 tablet_replica.cc:330] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: stopping tablet replica
I20260926 03:41:24.513703 17961 ts_tablet_manager.cc:1905] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:24.513774 17962 raft_consensus.cc:2229] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:24.514504 17962 raft_consensus.cc:2258] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:24.516228 17960 tablet_replica.cc:330] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: stopping tablet replica
I20260926 03:41:24.516808 17960 raft_consensus.cc:2229] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:41:24.517596 17960 raft_consensus.cc:2258] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:24.520092 17962 ts_tablet_manager.cc:1905] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:24.524096 17960 ts_tablet_manager.cc:1905] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:24.527688 17962 ts_tablet_manager.cc:1918] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:24.527946 17962 log.cc:1196] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/6e402aee165a4408b5f9835b0e352c67
I20260926 03:41:24.528510 17962 ts_tablet_manager.cc:1939] T 6e402aee165a4408b5f9835b0e352c67 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting consensus metadata
I20260926 03:41:24.530622 17445 catalog_manager.cc:4841] TS 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329): tablet 6e402aee165a4408b5f9835b0e352c67 (table test-table [id=62a4c8fdb56b4297a781b7bb3e247660]) successfully deleted
I20260926 03:41:24.535302 17961 ts_tablet_manager.cc:1918] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:24.535640 17961 log.cc:1196] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/6e402aee165a4408b5f9835b0e352c67
I20260926 03:41:24.536228 17961 ts_tablet_manager.cc:1939] T 6e402aee165a4408b5f9835b0e352c67 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting consensus metadata
I20260926 03:41:24.536885 17960 ts_tablet_manager.cc:1918] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:24.537140 17960 log.cc:1196] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/6e402aee165a4408b5f9835b0e352c67
I20260926 03:41:24.537789 17960 ts_tablet_manager.cc:1939] T 6e402aee165a4408b5f9835b0e352c67 P 9e9cf2b183274494bb665c023aeb4dae: Deleting consensus metadata
I20260926 03:41:24.538473 17448 catalog_manager.cc:4841] TS 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101): tablet 6e402aee165a4408b5f9835b0e352c67 (table test-table [id=62a4c8fdb56b4297a781b7bb3e247660]) successfully deleted
I20260926 03:41:24.540360 17447 catalog_manager.cc:4841] TS 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763): tablet 6e402aee165a4408b5f9835b0e352c67 (table test-table [id=62a4c8fdb56b4297a781b7bb3e247660]) successfully deleted
Sep 26 03:41:24 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:41:24.552101 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.501037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.1.221.129:41763)
0926 03:41:24.501244 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.501250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.512979 (+ 11729us) server_negotiation.cc:407] Connection header received
0926 03:41:24.515372 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.515419 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.515533 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.515644 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:24.518565 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.519797 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.528223 (+ 8426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.529093 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.529230 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.544150 (+ 14920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:24.544171 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:24.544193 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:24.544254 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:24.547729 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.549352 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.549362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.549365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.549576 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.550192 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.550196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.550198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.550702 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:24.550972 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.551568 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.551790 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:41:24.553083 17645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:24.660964 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17522
I20260926 03:41:24.679988 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17656
Sep 26 03:41:24 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:41:24.700729 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.681089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.1.221.130:45101)
0926 03:41:24.681335 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.681341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.681359 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:41:24.681430 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.681434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.681525 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.681621 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:24.683247 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.684268 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.685828 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.686272 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.686398 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.692407 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:24.692435 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:24.692459 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:24.692521 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:24.696534 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.697753 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.697764 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.697767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.698011 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.698589 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.698594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.698597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.699187 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:24.699588 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.700061 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.700354 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:41:24.701683 17779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:24.796563 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17656
I20260926 03:41:24.814088 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17790
Sep 26 03:41:24 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:41:24.831648 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:24.815036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.1.221.131:44329)
0926 03:41:24.815285 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:24.815291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:24.815314 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:41:24.815499 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:24.815504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:24.815576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:24.815694 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:24.817086 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.817829 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.819330 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:24.819837 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:24.819958 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:24.825294 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:24.825312 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:24.825331 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:24.825375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:24.828599 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.829418 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.829426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.829428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.829574 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:24.829961 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:24.829965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:24.829967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:24.830353 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:24.830640 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:24.831035 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 03:41:24.831281 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:41:24.832666 17913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:24.929109 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17790
I20260926 03:41:24.948453 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17429
I20260926 03:41:24.950071 17491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:25.069376 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17429
I20260926 03:41:45.092128 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--webserver_interface=127.1.221.190
--webserver_port=36575
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:41:45.472888 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:45.473268 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:45.473382 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:45.482180 17968 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:41:45.482319 17968 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:41:45.482362 17968 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:41:45.482429 17968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:41:45.482522 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:45.482621 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:45.482690 17968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:41:45.482771 17968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:41:45.493595 17968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=36575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:45.495064 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:45.496590 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:45.523237 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:45.523461 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:45.523232 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:45.524511 17968 server_base.cc:1034] running on GCE node
I20260926 03:41:45.525616 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:45.527532 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:45.528813 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790394105528733 us; error 60 us; skew 500 ppm
I20260926 03:41:45.529300 17968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:45 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:45.533850 17968 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:41:45.535949 17968 webserver.cc:460] Webserver started at http://127.1.221.190:36575/ using document root <none> and password file <none>
I20260926 03:41:45.536836 17968 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:45.536968 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:45.545215 17968 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:41:45.549295 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:45.551481 17968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 03:41:45.551718 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "089ce4b5c5cf472087dc53a02679ac8d"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "1289f4f3ea071f7c11811d2730d0277e"
server_key_iv: "3a00e8faf8ae183503fad9b13c0c5bf3"
server_key_version: "encryptionkey@0"
I20260926 03:41:45.552975 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:45.573508 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:45.579778 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:45.580109 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:45.601579 17968 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40107
I20260926 03:41:45.601590 18035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40107 every 8 connection(s)
I20260926 03:41:45.603458 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:41:45.611986 18036 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:41:45.614055 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 17968
I20260926 03:41:45.615108 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:41763
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=45339
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:41:45.616581 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap starting.
I20260926 03:41:45.641023 18036 log.cc:824] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:45.654563 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:41:45.655175 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap complete.
I20260926 03:41:45.666535 18036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:45.667538 18036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Initialized, Role: FOLLOWER
I20260926 03:41:45.668188 18036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:45.668422 18036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:41:45.668536 18036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:41:45.668710 18036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:41:45.672855 18036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:45.673462 18036 leader_election.cc:302] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089ce4b5c5cf472087dc53a02679ac8d; no voters:
I20260926 03:41:45.674069 18036 leader_election.cc:288] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:41:45.674226 18040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:41:45.675526 18040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 LEADER]: Becoming Leader. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Running, Role: LEADER
I20260926 03:41:45.676110 18040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:41:45.676621 18036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:41:45.678603 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:41:45.678992 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:45.680691 18045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:41:45.680496 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089ce4b5c5cf472087dc53a02679ac8d. Latest consensus state: current_term: 2 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:41:45.681207 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:41:45.689666 18045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:41:45.691501 18045 catalog_manager.cc:1259] Loaded cluster ID: 957677ab246b4fe1b55d0869d27537ad
I20260926 03:41:45.691710 18045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:41:45.695986 18045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:41:45.697839 18045 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Loaded TSK: 0
I20260926 03:41:45.744956 18045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Generated new TSK 1
I20260926 03:41:45.745498 18045 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:41:45.989095 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:45.989395 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:45.989523 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:45.997648 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:45.997780 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:45.997936 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:41:46.009938 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:41763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=45339
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:46.011421 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:46.012840 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:46.029476 18064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:46.032194 18063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:46.032359 18066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:46.034277 18038 server_base.cc:1034] running on GCE node
I20260926 03:41:46.034973 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:46.036172 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:46.037621 18038 hybrid_clock.cc:648] HybridClock initialized: now 1790394106037407 us; error 181 us; skew 500 ppm
I20260926 03:41:46.038100 18038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:46 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:46.048130 18038 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:41:46.049980 18038 webserver.cc:460] Webserver started at http://127.1.221.129:45339/ using document root <none> and password file <none>
I20260926 03:41:46.050714 18038 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:46.050858 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:46.056931 18038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:41:46.060778 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.062407 18038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:41:46.062592 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "9e9cf2b183274494bb665c023aeb4dae"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "d04e4befcba325ccec4359d04788e22c"
server_key_iv: "93acaa6ebe4fd0be077c0ba7d4e18d97"
server_key_version: "encryptionkey@0"
I20260926 03:41:46.063704 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:46.085596 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:46.090258 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:46.090538 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:46.091624 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:46.093393 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:46.093487 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.093602 18038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:46.093667 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.127796 18038 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41763
I20260926 03:41:46.127837 18185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41763 every 8 connection(s)
I20260926 03:41:46.129215 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:41:46.134924 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18038
I20260926 03:41:46.135730 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:45101
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=42791
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:41:46 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:46.154476 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:46.132251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:49755)
0926 03:41:46.133069 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:46.133113 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:46.134909 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:46.136583 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:46.136595 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:46.137170 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:46.137717 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:46.137735 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.139910 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.139914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:46.141190 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:46.141204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.142550 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.142555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:46.142791 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:46.143596 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:46.143634 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:46.146290 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:46.149845 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:46.149855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:46.149873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:46.150310 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:46.150757 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:46.150762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:46.150766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:46.150957 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:46.151762 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:46.151771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:46.152107 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 03:41:46.152370 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:41:46.153122 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":324,"threads_started":1}
I20260926 03:41:46.154487 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:46.132041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49755 (local address 127.1.221.190:40107)
0926 03:41:46.132616 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:46.132628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:46.133251 (+ 623us) server_negotiation.cc:407] Connection header received
0926 03:41:46.135567 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:46.135586 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:46.136043 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:46.136434 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:46.137930 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.139687 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.141348 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.142289 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.142478 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:46.146626 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:46.146635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:46.146650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:46.146696 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:46.149671 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:46.150461 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:46.150468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:46.150473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:46.150632 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:46.151066 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:46.151070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:46.151072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:46.151618 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:46.151807 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:46.152579 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 03:41:46.153191 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260926 03:41:46.157886 18186 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:46.158206 18186 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:46.159011 18186 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
I20260926 03:41:46.161701 18001 ts_manager.cc:194] Registered new tserver with Master: 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763)
I20260926 03:41:46.163251 18001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:49755
W20260926 03:41:46.481925 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:46.482245 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:46.482384 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:46.489842 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:46.489996 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:46.490154 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:41:46.500222 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:45101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=42791
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:46.501566 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:46.502894 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:46.516006 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:46.516072 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:46.516036 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:46.517232 18190 server_base.cc:1034] running on GCE node
I20260926 03:41:46.517904 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:46.518945 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:46.520164 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790394106520099 us; error 57 us; skew 500 ppm
I20260926 03:41:46.520498 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:46 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:46.523710 18190 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:41:46.525236 18190 webserver.cc:460] Webserver started at http://127.1.221.130:42791/ using document root <none> and password file <none>
I20260926 03:41:46.525926 18190 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:46.526055 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:46.531838 18190 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:41:46.534660 18205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.536140 18190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:41:46.536305 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "4d23671137764c5c8b9d6e288d4d50bf"
format_stamp: "Formatted at 2026-09-26 03:41:23 on dist-test-slave-lqrk"
server_key: "e71dc4d22ab27ee39f404079899b52b7"
server_key_iv: "0b1702e395c7f91afd392e82a0dbd914"
server_key_version: "encryptionkey@0"
I20260926 03:41:46.537313 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:46.590606 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:46.595934 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:46.596207 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:46.597219 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:46.598605 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:46.598701 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.598811 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:46.598881 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:46.637336 18190 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45101
I20260926 03:41:46.637378 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45101 every 8 connection(s)
I20260926 03:41:46.639863 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:41:46.640255 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18190
I20260926 03:41:46.640981 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:44329
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=39765
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:41:46 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:46.663793 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:46.642865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35487 (local address 127.1.221.190:40107)
0926 03:41:46.643063 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:46.643067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:46.644550 (+ 1483us) server_negotiation.cc:407] Connection header received
0926 03:41:46.646526 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:46.646532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:46.646635 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:46.646722 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:46.648465 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.649193 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.651473 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.651978 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.652104 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:46.656613 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:46.656620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:46.656623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:46.656682 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:46.659852 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:46.661063 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:46.661068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:46.661070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:46.661215 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:46.661795 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:46.661799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:46.661801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:46.662127 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:46.662244 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:46.663247 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0926 03:41:46.663533 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:41:46.665571 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:46.643559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:35487)
0926 03:41:46.644358 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:46.644421 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:46.646208 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:46.646990 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:46.646999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:46.647645 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:46.648248 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:46.648273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.649355 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.649360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:46.651252 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:46.651268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:46.652251 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:46.652258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:46.652562 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:46.653323 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:46.653359 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:46.656297 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:46.660078 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:46.660092 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:46.660115 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:46.660889 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:46.661333 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:46.661338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:46.661342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:46.661661 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:46.662261 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:46.662271 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:46.662803 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 03:41:46.663149 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 03:41:46.664016 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":321,"threads_started":1}
I20260926 03:41:46.667222 18318 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:46.667562 18318 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:46.668344 18318 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
I20260926 03:41:46.669867 18001 ts_manager.cc:194] Registered new tserver with Master: 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101)
I20260926 03:41:46.670718 18001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:35487
W20260926 03:41:46.983685 18321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:41:46.984002 18321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:41:46.984133 18321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:41:46.991479 18321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:41:46.991624 18321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:41:46.991771 18321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:41:47.001665 18321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:44329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=39765
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:41:47.002993 18321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:41:47.004415 18321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:41:47.017525 18326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:47.017686 18329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:41:47.017810 18327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:41:47.018411 18321 server_base.cc:1034] running on GCE node
I20260926 03:41:47.019060 18321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:41:47.020076 18321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:41:47.021284 18321 hybrid_clock.cc:648] HybridClock initialized: now 1790394107021223 us; error 47 us; skew 500 ppm
I20260926 03:41:47.021595 18321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:41:47 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:41:47.024900 18321 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:41:47.026413 18321 webserver.cc:460] Webserver started at http://127.1.221.131:39765/ using document root <none> and password file <none>
I20260926 03:41:47.027124 18321 fs_manager.cc:359] Metadata directory not provided
I20260926 03:41:47.027257 18321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:41:47.033614 18321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 03:41:47.036917 18336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:47.038455 18321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:41:47.038618 18321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "54ec71fa785b48b69a8ed8285859bb5f"
format_stamp: "Formatted at 2026-09-26 03:41:24 on dist-test-slave-lqrk"
server_key: "2d7f97e0cc8d129b78361aa5bb6d0f0a"
server_key_iv: "588c2477fee0975f37a8804027a92232"
server_key_version: "encryptionkey@0"
I20260926 03:41:47.039710 18321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:41:47.063889 18321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:41:47.068750 18321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:41:47.069043 18321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:41:47.070556 18321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:41:47.072566 18321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:41:47.072679 18321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:47.072788 18321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:41:47.072858 18321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:41:47.104671 18321 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44329
I20260926 03:41:47.104718 18448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44329 every 8 connection(s)
I20260926 03:41:47.106299 18321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:41:47.109232 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18321
Sep 26 03:41:47 dist-test-slave-lqrk krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:41:47.130489 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.110182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.1.221.190:40107)
0926 03:41:47.114115 (+ 3933us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.114120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.114135 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:41:47.114177 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.114180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.114231 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.114312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:47.116374 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.117444 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.119616 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.120205 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.120346 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.123852 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:47.123861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:47.123867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:47.123911 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:47.127312 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.128241 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.128246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.128249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.128401 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.128778 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.128783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.128787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.129083 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:47.129335 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.129862 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.130091 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3834,"thread_start_us":3723,"threads_started":1}
I20260926 03:41:47.132166 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.108998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:43463 (local address 127.1.221.190:40107)
0926 03:41:47.109235 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.109241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.110544 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 03:41:47.113696 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.113702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.113787 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.113957 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:47.116153 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.117322 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.119846 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.120478 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.120596 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.125077 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:47.125085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:47.125090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:47.125148 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:47.128204 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.129181 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.129186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.129189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.129321 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.129840 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.129846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.129849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.130197 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:47.130513 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.131457 (+ 944us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.131812 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:41:47.133246 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.109505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:43463)
0926 03:41:47.110359 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.110414 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.113292 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.114223 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.114233 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.115026 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:41:47.115934 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.115961 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.117548 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.117555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.119707 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.119722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.120736 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.120744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.121045 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.121940 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:41:47.121975 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:41:47.124791 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:41:47.128372 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:47.128383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:47.128402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:47.129047 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:47.129432 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:41:47.129437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:41:47.129440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:41:47.129680 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:41:47.130609 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:41:47.130616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:41:47.131027 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.131339 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.131919 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":325,"threads_started":1}
I20260926 03:41:47.134529 18449 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40107
I20260926 03:41:47.134790 18449 heartbeater.cc:464] Registering TS with master...
I20260926 03:41:47.135540 18449 heartbeater.cc:511] Master 127.1.221.190:40107 requested a full tablet report, sending...
I20260926 03:41:47.136960 18001 ts_manager.cc:194] Registered new tserver with Master: 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:47.137796 18001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:43463
I20260926 03:41:47.145479 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:41:47.152141 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.146445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.1.221.190:40107)
0926 03:41:47.146664 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.146671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.146973 (+ 302us) server_negotiation.cc:407] Connection header received
0926 03:41:47.147140 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.147144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.147298 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.147477 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:41:47.147912 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.148676 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.150273 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.150707 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.150828 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.151359 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:41:47.151623 (+ 264us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 03:41:47.151933 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41078: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 03:41:47.152280 18188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41078: token expired
W20260926 03:41:47.152946 17925 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.221.190:40107)
I20260926 03:41:47.165824 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.153411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.1.221.190:40107)
0926 03:41:47.153585 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.153591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.153609 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:41:47.153834 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.153838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.153916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.154002 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:47.155357 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.156154 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.157519 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.157962 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.158077 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.160759 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:47.160765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:47.160768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:47.160804 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:47.163233 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.163939 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.163945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.163949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.164077 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.164415 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.164420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.164424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.164752 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:47.164986 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.165296 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.165526 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:41:47.166730 18186 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
I20260926 03:41:47.169720 18000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:41:47.177219 18000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:41:47.204882 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.190511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.1.221.130:45101)
0926 03:41:47.195972 (+ 5461us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.195983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.196023 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:41:47.196151 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.196161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.196495 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.197627 (+ 1132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:47.198126 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199797 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.202512 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.203690 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.203824 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.204062 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.204377 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.204548 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5279,"thread_start_us":5133,"threads_started":1}
I20260926 03:41:47.204982 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.190437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:45101 (local address 127.0.0.1:49138)
0926 03:41:47.194819 (+ 4382us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.194841 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.194972 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.196802 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.196810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.196843 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:47.197369 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.197385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199966 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.199971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.202299 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.202314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.203917 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.203925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.204155 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.204195 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.204314 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.204524 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4233,"mutex_wait_us":137,"thread_start_us":5671,"threads_started":1}
I20260926 03:41:47.205513 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.189877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41763 (local address 127.0.0.1:41164)
0926 03:41:47.191580 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.191632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.191821 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.198086 (+ 6265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.198090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.198114 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:47.198448 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.198458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199916 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.199921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.202651 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.202660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.204665 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.204672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.204851 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.204888 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.204959 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.205096 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"mutex_wait_us":134,"thread_start_us":2143,"threads_started":1}
I20260926 03:41:47.206322 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.190008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.1.221.129:41763)
0926 03:41:47.197526 (+ 7518us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.197534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.197560 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:41:47.197645 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.197652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.197879 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.198597 (+ 718us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:47.198769 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199775 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.202840 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.204408 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.205521 (+ 1113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.205760 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.205842 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.206022 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6948,"thread_start_us":6771,"threads_started":1}
I20260926 03:41:47.206868 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.190498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44329 (local address 127.0.0.1:45786)
0926 03:41:47.196382 (+ 5884us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.196398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.196533 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.197135 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.197158 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.197185 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:47.197603 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.197616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199946 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.199950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.202641 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.202658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.204934 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.204940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.205150 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.205183 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.205268 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.206520 (+ 1252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5755,"mutex_wait_us":102,"thread_start_us":2294,"threads_started":1}
I20260926 03:41:47.207930 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.191135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.1.221.131:44329)
0926 03:41:47.195656 (+ 4521us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.195667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.196505 (+ 838us) server_negotiation.cc:407] Connection header received
0926 03:41:47.196668 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.196676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.196930 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.197132 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:47.197972 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.199782 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.202854 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.204400 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.204539 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.204752 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.207367 (+ 2615us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.207585 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4326,"thread_start_us":4123,"threads_started":1}
I20260926 03:41:47.209650 18253 tablet_service.cc:1467] Processing CreateTablet for tablet b7832029fdbb45559aed40e56c94ce11 (DEFAULT_TABLE table=test-table [id=5fec96511d40446c849087afe47f7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:47.209743 18121 tablet_service.cc:1467] Processing CreateTablet for tablet b7832029fdbb45559aed40e56c94ce11 (DEFAULT_TABLE table=test-table [id=5fec96511d40446c849087afe47f7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:47.210947 18253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7832029fdbb45559aed40e56c94ce11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:47.211258 18121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7832029fdbb45559aed40e56c94ce11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:47.211735 18384 tablet_service.cc:1467] Processing CreateTablet for tablet b7832029fdbb45559aed40e56c94ce11 (DEFAULT_TABLE table=test-table [id=5fec96511d40446c849087afe47f7e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:41:47.213186 18384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7832029fdbb45559aed40e56c94ce11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:41:47.224541 18465 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Bootstrap starting.
I20260926 03:41:47.226933 18466 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Bootstrap starting.
I20260926 03:41:47.228698 18467 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Bootstrap starting.
I20260926 03:41:47.228729 18465 tablet_bootstrap.cc:654] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:47.230206 18465 log.cc:824] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:47.230548 18466 tablet_bootstrap.cc:654] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:47.231886 18467 tablet_bootstrap.cc:654] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Neither blocks nor log segments found. Creating new log.
I20260926 03:41:47.231866 18466 log.cc:824] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:47.233011 18467 log.cc:824] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:41:47.235190 18466 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: No bootstrap required, opened a new log
I20260926 03:41:47.235179 18465 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: No bootstrap required, opened a new log
I20260926 03:41:47.235359 18467 tablet_bootstrap.cc:492] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: No bootstrap required, opened a new log
I20260926 03:41:47.235509 18466 ts_tablet_manager.cc:1397] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:41:47.235505 18465 ts_tablet_manager.cc:1397] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 03:41:47.235639 18467 ts_tablet_manager.cc:1397] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260926 03:41:47.243690 18467 raft_consensus.cc:348] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.244150 18467 raft_consensus.cc:374] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:47.244244 18467 raft_consensus.cc:729] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ec71fa785b48b69a8ed8285859bb5f, State: Initialized, Role: FOLLOWER
I20260926 03:41:47.245021 18467 consensus_queue.cc:260] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.246150 18449 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
I20260926 03:41:47.246228 18465 raft_consensus.cc:348] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.246675 18465 raft_consensus.cc:374] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:47.246773 18465 raft_consensus.cc:729] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d23671137764c5c8b9d6e288d4d50bf, State: Initialized, Role: FOLLOWER
I20260926 03:41:47.246879 18467 ts_tablet_manager.cc:1428] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260926 03:41:47.247424 18465 consensus_queue.cc:260] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.247747 18466 raft_consensus.cc:348] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.248193 18466 raft_consensus.cc:374] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:41:47.248281 18466 raft_consensus.cc:729] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9cf2b183274494bb665c023aeb4dae, State: Initialized, Role: FOLLOWER
I20260926 03:41:47.248838 18466 consensus_queue.cc:260] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.249190 18465 ts_tablet_manager.cc:1428] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 03:41:47.250345 18318 heartbeater.cc:503] Master 127.1.221.190:40107 was elected leader, sending a full tablet report...
I20260926 03:41:47.250557 18466 ts_tablet_manager.cc:1428] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260926 03:41:47.358071 18450 tablet.cc:2369] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:41:47.381596 18187 tablet.cc:2369] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:41:47.392292 18319 tablet.cc:2369] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:41:47.630312 18472 raft_consensus.cc:482] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:41:47.630594 18472 raft_consensus.cc:504] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.632305 18472 leader_election.cc:288] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763), 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:47.639989 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.632772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36607 (local address 127.1.221.129:41763)
0926 03:41:47.632969 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.632975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.633991 (+ 1016us) server_negotiation.cc:407] Connection header received
0926 03:41:47.634153 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.634158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.634218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.634299 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:47.634878 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.635649 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.637669 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.638883 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.639060 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.639135 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.639587 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.639730 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:41:47.639989 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.632828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:48141 (local address 127.1.221.131:44329)
0926 03:41:47.633004 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.633010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.633576 (+ 566us) server_negotiation.cc:407] Connection header received
0926 03:41:47.633850 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.633857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.633941 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.634000 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:41:47.634619 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.635305 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.637742 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.638937 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.639070 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.639135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.639627 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.639755 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:41:47.640089 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.633126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41763 (local address 127.1.221.130:36607)
0926 03:41:47.633882 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.633897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.634025 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.634360 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.634364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.634392 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:47.634730 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.634738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.635778 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.635783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.637505 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.637520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.639142 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.639149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.639356 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.639418 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.639527 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.639671 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":46,"thread_start_us":434,"threads_started":1}
I20260926 03:41:47.640852 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.632756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44329 (local address 127.1.221.130:48141)
0926 03:41:47.633455 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 03:41:47.633485 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:41:47.633644 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:41:47.634166 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:41:47.634169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:41:47.634196 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:41:47.634478 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.634490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.635470 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.635474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.637577 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:41:47.637593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.639282 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.639287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:41:47.639442 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.639465 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:41:47.640294 (+ 829us) client_negotiation.cc:240] Negotiation successful
0926 03:41:47.640427 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":45,"thread_start_us":370,"threads_started":1}
I20260926 03:41:47.641201 18141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7832029fdbb45559aed40e56c94ce11" candidate_uuid: "4d23671137764c5c8b9d6e288d4d50bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9cf2b183274494bb665c023aeb4dae" is_pre_election: true
I20260926 03:41:47.642215 18141 raft_consensus.cc:2454] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d23671137764c5c8b9d6e288d4d50bf in term 0.
I20260926 03:41:47.642454 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7832029fdbb45559aed40e56c94ce11" candidate_uuid: "4d23671137764c5c8b9d6e288d4d50bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ec71fa785b48b69a8ed8285859bb5f" is_pre_election: true
I20260926 03:41:47.643009 18404 raft_consensus.cc:2454] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d23671137764c5c8b9d6e288d4d50bf in term 0.
I20260926 03:41:47.643271 18208 leader_election.cc:302] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d23671137764c5c8b9d6e288d4d50bf, 9e9cf2b183274494bb665c023aeb4dae; no voters:
I20260926 03:41:47.643785 18472 raft_consensus.cc:2781] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:41:47.643935 18472 raft_consensus.cc:482] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:41:47.644054 18472 raft_consensus.cc:3037] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:47.646967 18472 raft_consensus.cc:504] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.647694 18472 leader_election.cc:288] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [CANDIDATE]: Term 1 election: Requested vote from peers 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763), 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329)
I20260926 03:41:47.648237 18141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7832029fdbb45559aed40e56c94ce11" candidate_uuid: "4d23671137764c5c8b9d6e288d4d50bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9cf2b183274494bb665c023aeb4dae"
I20260926 03:41:47.648461 18141 raft_consensus.cc:3037] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:47.648514 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7832029fdbb45559aed40e56c94ce11" candidate_uuid: "4d23671137764c5c8b9d6e288d4d50bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ec71fa785b48b69a8ed8285859bb5f"
I20260926 03:41:47.648849 18404 raft_consensus.cc:3037] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:41:47.650826 18141 raft_consensus.cc:2454] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d23671137764c5c8b9d6e288d4d50bf in term 1.
I20260926 03:41:47.651311 18208 leader_election.cc:302] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d23671137764c5c8b9d6e288d4d50bf, 9e9cf2b183274494bb665c023aeb4dae; no voters:
I20260926 03:41:47.651690 18472 raft_consensus.cc:2781] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:41:47.652729 18472 raft_consensus.cc:686] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 LEADER]: Becoming Leader. State: Replica: 4d23671137764c5c8b9d6e288d4d50bf, State: Running, Role: LEADER
I20260926 03:41:47.652814 18404 raft_consensus.cc:2454] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d23671137764c5c8b9d6e288d4d50bf in term 1.
I20260926 03:41:47.653574 18472 consensus_queue.cc:237] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } }
I20260926 03:41:47.659250 17999 catalog_manager.cc:5495] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130). New cstate: current_term: 1 leader_uuid: "4d23671137764c5c8b9d6e288d4d50bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d23671137764c5c8b9d6e288d4d50bf" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 45101 } health_report { overall_health: HEALTHY } } }
I20260926 03:41:47.771823 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.764852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.1.221.130:45101)
0926 03:41:47.765386 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.765393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.765416 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:41:47.765546 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.765551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.765666 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.765882 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:41:47.766394 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.767301 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.768974 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.769432 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.769548 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.770155 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:41:47.770995 (+ 840us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:41:47.771143 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.771316 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.771478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
I20260926 03:41:47.781064 18141 raft_consensus.cc:1261] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Refusing update from remote peer 4d23671137764c5c8b9d6e288d4d50bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:47.781200 18404 raft_consensus.cc:1261] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Refusing update from remote peer 4d23671137764c5c8b9d6e288d4d50bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:41:47.782320 18472 consensus_queue.cc:1035] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9cf2b183274494bb665c023aeb4dae" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:41:47.782685 18476 consensus_queue.cc:1035] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ec71fa785b48b69a8ed8285859bb5f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:41:47.814206 17999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:47.814595 17999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:41:47.819046 17999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Sending DeleteTablet for 3 replicas of tablet b7832029fdbb45559aed40e56c94ce11
I20260926 03:41:47.820705 18121 tablet_service.cc:1514] Processing DeleteTablet for tablet b7832029fdbb45559aed40e56c94ce11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41164
I20260926 03:41:47.820765 18253 tablet_service.cc:1514] Processing DeleteTablet for tablet b7832029fdbb45559aed40e56c94ce11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:49138
I20260926 03:41:47.821133 18384 tablet_service.cc:1514] Processing DeleteTablet for tablet b7832029fdbb45559aed40e56c94ce11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:41:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:45786
I20260926 03:41:47.822557 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18038
I20260926 03:41:47.833531 18492 tablet_replica.cc:330] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: stopping tablet replica
I20260926 03:41:47.834386 18494 tablet_replica.cc:330] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: stopping tablet replica
I20260926 03:41:47.834852 18492 raft_consensus.cc:2229] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:47.834899 18494 raft_consensus.cc:2229] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:41:47.835582 18492 raft_consensus.cc:2258] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:47.835582 18494 raft_consensus.cc:2258] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:47.838187 18492 ts_tablet_manager.cc:1905] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:47.838195 18494 ts_tablet_manager.cc:1905] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:47.846508 18494 ts_tablet_manager.cc:1918] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:47.846508 18492 ts_tablet_manager.cc:1918] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:47.846851 18492 log.cc:1196] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/b7832029fdbb45559aed40e56c94ce11
I20260926 03:41:47.847025 18494 log.cc:1196] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/b7832029fdbb45559aed40e56c94ce11
I20260926 03:41:47.847518 18492 ts_tablet_manager.cc:1939] T b7832029fdbb45559aed40e56c94ce11 P 9e9cf2b183274494bb665c023aeb4dae: Deleting consensus metadata
I20260926 03:41:47.847854 18494 ts_tablet_manager.cc:1939] T b7832029fdbb45559aed40e56c94ce11 P 54ec71fa785b48b69a8ed8285859bb5f: Deleting consensus metadata
I20260926 03:41:47.850057 17987 catalog_manager.cc:4841] TS 9e9cf2b183274494bb665c023aeb4dae (127.1.221.129:41763): tablet b7832029fdbb45559aed40e56c94ce11 (table test-table [id=5fec96511d40446c849087afe47f7e1b]) successfully deleted
I20260926 03:41:47.850207 17985 catalog_manager.cc:4841] TS 54ec71fa785b48b69a8ed8285859bb5f (127.1.221.131:44329): tablet b7832029fdbb45559aed40e56c94ce11 (table test-table [id=5fec96511d40446c849087afe47f7e1b]) successfully deleted
I20260926 03:41:47.851118 18493 tablet_replica.cc:330] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: stopping tablet replica
I20260926 03:41:47.851756 18493 raft_consensus.cc:2229] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:41:47.852622 18493 raft_consensus.cc:2258] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:41:47.855065 18493 ts_tablet_manager.cc:1905] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:41:47.863605 18493 ts_tablet_manager.cc:1918] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:41:47.863899 18493 log.cc:1196] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/b7832029fdbb45559aed40e56c94ce11
I20260926 03:41:47.864573 18493 ts_tablet_manager.cc:1939] T b7832029fdbb45559aed40e56c94ce11 P 4d23671137764c5c8b9d6e288d4d50bf: Deleting consensus metadata
I20260926 03:41:47.866969 17988 catalog_manager.cc:4841] TS 4d23671137764c5c8b9d6e288d4d50bf (127.1.221.130:45101): tablet b7832029fdbb45559aed40e56c94ce11 (table test-table [id=5fec96511d40446c849087afe47f7e1b]) successfully deleted
I20260926 03:41:47.868512 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.824047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.1.221.129:41763)
0926 03:41:47.824300 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.824307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.847796 (+ 23489us) server_negotiation.cc:407] Connection header received
0926 03:41:47.848111 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.848118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.848235 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.848319 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:47.850597 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.851740 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.853906 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.854603 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.854760 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:47.859108 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:47.859132 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:47.859155 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:47.859215 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:47.863622 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.865188 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.865200 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.865204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.865456 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:47.866014 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:47.866020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:47.866023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:47.866581 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:47.866926 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:47.867704 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 03:41:47.867999 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:41:47.869410 18181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:47.973304 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18038
I20260926 03:41:47.993119 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18190
I20260926 03:41:48.013188 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:47.994182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.1.221.130:45101)
0926 03:41:47.994453 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:47.994459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:47.994487 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:41:47.994685 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:47.994691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:47.994789 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:47.994932 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:47.996674 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:47.997837 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:47.999915 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:48.000626 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:48.000786 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:48.004153 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:48.004173 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:48.004195 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:48.004252 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:48.008676 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:48.009821 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:48.009832 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:48.009835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:48.010043 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:48.010620 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:48.010626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:48.010630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:48.011270 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:48.011883 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:48.012433 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 03:41:48.012743 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 03:41:48.014238 18313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:48.162817 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18190
I20260926 03:41:48.184798 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18321
I20260926 03:41:48.203675 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:41:48.185803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.1.221.131:44329)
0926 03:41:48.186481 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0926 03:41:48.186487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:41:48.186510 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:41:48.186597 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:41:48.186602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:41:48.186682 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:41:48.186801 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:41:48.188469 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:48.189492 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:48.191203 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:41:48.191857 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:41:48.192003 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:41:48.195498 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:41:48.195580 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:41:48.195605 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:41:48.195665 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:41:48.199326 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:48.200382 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:48.200396 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:48.200399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:48.200754 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:41:48.201293 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:41:48.201298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:41:48.201301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:41:48.201842 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:41:48.202191 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:41:48.202775 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 03:41:48.203126 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":391,"threads_started":1}
I20260926 03:41:48.204854 18444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:48.302060 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18321
I20260926 03:41:48.320007 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 17968
I20260926 03:41:48.321657 18031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:41:48.451323 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 17968
I20260926 03:42:08.477753 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--webserver_interface=127.1.221.190
--webserver_port=36575
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:08.908577 18501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:08.909029 18501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:08.909168 18501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:08.917961 18501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:42:08.918123 18501 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:42:08.918170 18501 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:42:08.918267 18501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:08.918358 18501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:08.918465 18501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:08.918530 18501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:08.918584 18501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:08.930576 18501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=36575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:08.932444 18501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:08.934154 18501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:08.949760 18506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:08.949713 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:08.949811 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:08.950776 18501 server_base.cc:1034] running on GCE node
I20260926 03:42:08.951958 18501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:08.953982 18501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:08.955339 18501 hybrid_clock.cc:648] HybridClock initialized: now 1790394128955191 us; error 75 us; skew 500 ppm
I20260926 03:42:08.955891 18501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:08 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:08.966130 18501 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:42:08.968568 18501 webserver.cc:460] Webserver started at http://127.1.221.190:36575/ using document root <none> and password file <none>
I20260926 03:42:08.969638 18501 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:08.969789 18501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:08.977622 18501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:42:08.981948 18516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:08.984248 18501 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 03:42:08.984524 18501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "089ce4b5c5cf472087dc53a02679ac8d"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "1289f4f3ea071f7c11811d2730d0277e"
server_key_iv: "3a00e8faf8ae183503fad9b13c0c5bf3"
server_key_version: "encryptionkey@0"
I20260926 03:42:08.986091 18501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:09.032874 18501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:09.039312 18501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:09.039770 18501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:09.063025 18501 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40107
I20260926 03:42:09.063098 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40107 every 8 connection(s)
I20260926 03:42:09.065094 18501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:09.070688 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18501
I20260926 03:42:09.071839 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:41763
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=45339
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:42:09.074383 18568 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:42:09.079276 18568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap starting.
I20260926 03:42:09.102640 18568 log.cc:824] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:09.121093 18568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:42:09.121685 18568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Bootstrap complete.
I20260926 03:42:09.132836 18568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:42:09.133846 18568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Initialized, Role: FOLLOWER
I20260926 03:42:09.134634 18568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:42:09.134873 18568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:09.135008 18568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:09.135176 18568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:42:09.139703 18568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:42:09.140463 18568 leader_election.cc:302] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089ce4b5c5cf472087dc53a02679ac8d; no voters:
I20260926 03:42:09.141436 18568 leader_election.cc:288] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:42:09.141709 18572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:42:09.143057 18572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [term 3 LEADER]: Becoming Leader. State: Replica: 089ce4b5c5cf472087dc53a02679ac8d, State: Running, Role: LEADER
I20260926 03:42:09.143961 18572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } }
I20260926 03:42:09.146045 18568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:09.147316 18574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089ce4b5c5cf472087dc53a02679ac8d. Latest consensus state: current_term: 3 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:42:09.147900 18574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:09.147276 18573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "089ce4b5c5cf472087dc53a02679ac8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089ce4b5c5cf472087dc53a02679ac8d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40107 } } }
I20260926 03:42:09.148638 18573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:09.149098 18578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:09.165763 18578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:09.167649 18578 catalog_manager.cc:1259] Loaded cluster ID: 957677ab246b4fe1b55d0869d27537ad
I20260926 03:42:09.167866 18578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:09.175892 18578 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:09.178125 18578 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Loaded TSK: 0
I20260926 03:42:09.178579 18578 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 089ce4b5c5cf472087dc53a02679ac8d: Loaded TSK: 1
F20260926 03:42:09.180038 18578 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fafa8a7d9ad google::LogMessage::Fail() at ??:0
@ 0x7fafa8a8088a google::LogMessage::SendToLog() at ??:0
@ 0x7fafa8a7d4de google::LogMessage::Flush() at ??:0
@ 0x7fafa8a7ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fafac7e17d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fafac7d073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fafbe4b21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fafbe4b1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fafbe53d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fafbe53d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fafbe5875e9 std::function<>::operator()() at ??:0
@ 0x7fafbe4b5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fafbe4b3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fafbe53b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fafbe53b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fafab6c9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fafab6cc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fafab6cc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fafab6a7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fafae9816db start_thread at ??:0
@ 0x7fafa791761f clone at ??:0
W20260926 03:42:09.558481 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:09.558789 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:09.558887 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:09.566569 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:09.566696 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:09.566811 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:09.577978 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:41763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=45339
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:09.579340 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:09.580678 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:09.594929 18615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:09.594887 18618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:09.594946 18616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:09.595674 18570 server_base.cc:1034] running on GCE node
I20260926 03:42:09.596303 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:09.597325 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:09.598553 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790394129598486 us; error 58 us; skew 500 ppm
I20260926 03:42:09.598908 18570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:09 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394129, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:09.602504 18570 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:42:09.604333 18570 webserver.cc:460] Webserver started at http://127.1.221.129:45339/ using document root <none> and password file <none>
I20260926 03:42:09.605088 18570 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:09.605194 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:09.611032 18570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:42:09.614060 18625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:09.615540 18570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:42:09.615705 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "9e9cf2b183274494bb665c023aeb4dae"
format_stamp: "Formatted at 2026-09-26 03:41:22 on dist-test-slave-lqrk"
server_key: "d04e4befcba325ccec4359d04788e22c"
server_key_iv: "93acaa6ebe4fd0be077c0ba7d4e18d97"
server_key_version: "encryptionkey@0"
I20260926 03:42:09.616778 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:09.636588 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:09.641573 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:09.641857 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:09.642860 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:09.644356 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:09.644454 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:09.644569 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:09.644639 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:09.679564 18570 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41763
I20260926 03:42:09.679620 18737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41763 every 8 connection(s)
I20260926 03:42:09.681077 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:09.685973 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:09.684100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:42481)
0926 03:42:09.684874 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:09.685113 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
W20260926 03:42:09.686775 18738 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40107 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:40107: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
I20260926 03:42:09.687690 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:09.687272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:49401)
0926 03:42:09.687500 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:09.687574 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:42:09.688443 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18570
I20260926 03:42:09.688694 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:09.688310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:39791)
0926 03:42:09.688457 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:09.688551 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:42:09.688970 18738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:42:09.689193 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:45101
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=42791
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:10.033041 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:10.033345 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:10.033473 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:10.041002 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:10.041128 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:10.041280 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:10.051319 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:45101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=42791
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:10.052771 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:10.054126 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:10.067621 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:10.067584 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:10.068084 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:10.068692 18741 server_base.cc:1034] running on GCE node
I20260926 03:42:10.069337 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:10.070358 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:10.071545 18741 hybrid_clock.cc:648] HybridClock initialized: now 1790394130071486 us; error 53 us; skew 500 ppm
I20260926 03:42:10.071852 18741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:10 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394130, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:10.075001 18741 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:42:10.076624 18741 webserver.cc:460] Webserver started at http://127.1.221.130:42791/ using document root <none> and password file <none>
I20260926 03:42:10.077322 18741 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:10.077443 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:10.083215 18741 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:42:10.086086 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.087555 18741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:10.087723 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "4d23671137764c5c8b9d6e288d4d50bf"
format_stamp: "Formatted at 2026-09-26 03:41:23 on dist-test-slave-lqrk"
server_key: "e71dc4d22ab27ee39f404079899b52b7"
server_key_iv: "0b1702e395c7f91afd392e82a0dbd914"
server_key_version: "encryptionkey@0"
I20260926 03:42:10.088740 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:10.103045 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:10.107417 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:10.107687 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:10.108891 18741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:10.110390 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:10.110489 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.110595 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:10.110662 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.146817 18741 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45101
I20260926 03:42:10.146885 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45101 every 8 connection(s)
I20260926 03:42:10.148331 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:10.153316 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.151418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:58793)
0926 03:42:10.152181 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.152428 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":270,"threads_started":1}
I20260926 03:42:10.154004 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18741
W20260926 03:42:10.154194 18869 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40107 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:40107: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
I20260926 03:42:10.154744 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:44329
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:40107
--webserver_port=39765
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:42:10.155114 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.154766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:48645)
0926 03:42:10.154929 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.155003 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:42:10.155992 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.155661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:55663)
0926 03:42:10.155800 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.155878 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:42:10.156222 18869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:42:10.505450 18872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:10.505757 18872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:10.505849 18872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:10.513396 18872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:10.513533 18872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:10.513649 18872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:10.524011 18872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:44329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=39765
--tserver_master_addrs=127.1.221.190:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:10.525302 18872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:10.526607 18872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:10.540148 18877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:10.540148 18878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:10.540148 18880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:10.540994 18872 server_base.cc:1034] running on GCE node
I20260926 03:42:10.541642 18872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:10.542634 18872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:10.543843 18872 hybrid_clock.cc:648] HybridClock initialized: now 1790394130543789 us; error 46 us; skew 500 ppm
I20260926 03:42:10.544190 18872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:10 dist-test-slave-lqrk krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394130, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:10.547292 18872 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:42:10.548825 18872 webserver.cc:460] Webserver started at http://127.1.221.131:39765/ using document root <none> and password file <none>
I20260926 03:42:10.549559 18872 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:10.549677 18872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:10.555542 18872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:42:10.558764 18887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.560379 18872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:10.560572 18872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "54ec71fa785b48b69a8ed8285859bb5f"
format_stamp: "Formatted at 2026-09-26 03:41:24 on dist-test-slave-lqrk"
server_key: "2d7f97e0cc8d129b78361aa5bb6d0f0a"
server_key_iv: "588c2477fee0975f37a8804027a92232"
server_key_version: "encryptionkey@0"
I20260926 03:42:10.561610 18872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:10.596689 18872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:10.601155 18872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:10.601429 18872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:10.602432 18872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:10.604058 18872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:10.604172 18872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.604297 18872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:10.604364 18872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:10.634512 18872 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44329
I20260926 03:42:10.634572 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44329 every 8 connection(s)
I20260926 03:42:10.636047 18872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:10.640399 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 18872
I20260926 03:42:10.641762 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.639317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:45779)
0926 03:42:10.640286 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.640625 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":312,"threads_started":1}
W20260926 03:42:10.642549 19000 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40107 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:40107: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
I20260926 03:42:10.643494 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.643060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:52981)
0926 03:42:10.643227 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.643313 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:42:10.645088 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.644529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:33429)
0926 03:42:10.644770 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.644896 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260926 03:42:10.645540 19000 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:42:10.690912 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:10.689987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:52013)
0926 03:42:10.690516 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:10.690646 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260926 03:42:11.157703 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:11.156831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:35493)
0926 03:42:11.157333 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:11.157457 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
I20260926 03:42:11.647550 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:11.646448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:37585)
0926 03:42:11.647101 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:11.647224 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":316,"threads_started":1}
I20260926 03:42:11.693248 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:11.692031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:33253)
0926 03:42:11.692861 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:11.693002 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":325,"threads_started":1}
I20260926 03:42:12.159896 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:12.158900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:32783)
0926 03:42:12.159542 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:12.159665 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":263,"threads_started":1}
I20260926 03:42:12.649905 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:12.648808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:52057)
0926 03:42:12.649509 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:12.649631 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":358,"threads_started":1}
I20260926 03:42:12.695664 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:12.694613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:51953)
0926 03:42:12.695257 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:12.695432 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
I20260926 03:42:13.161926 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:13.160966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:48709)
0926 03:42:13.161577 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:13.161695 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":314,"threads_started":1}
I20260926 03:42:13.652112 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:13.651226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:45227)
0926 03:42:13.651778 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:13.651875 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":294,"threads_started":1}
I20260926 03:42:13.698060 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:13.697061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:33619)
0926 03:42:13.697700 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:13.697810 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":337,"threads_started":1}
I20260926 03:42:14.164223 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:14.163135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:40041)
0926 03:42:14.163890 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:14.164005 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":427,"threads_started":1}
I20260926 03:42:14.654526 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:14.653445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:56693)
0926 03:42:14.654216 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:14.654336 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":367,"threads_started":1}
I20260926 03:42:14.700291 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:14.699304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:49681)
0926 03:42:14.699970 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:14.700084 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":332,"threads_started":1}
I20260926 03:42:15.166550 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:15.165613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:46863)
0926 03:42:15.166251 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:15.166359 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260926 03:42:15.656824 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:15.655862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:45293)
0926 03:42:15.656541 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:15.656648 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":367,"threads_started":1}
I20260926 03:42:15.702433 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:15.701481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:36607)
0926 03:42:15.702117 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:15.702235 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":338,"threads_started":1}
I20260926 03:42:16.168778 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:16.167899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:35533)
0926 03:42:16.168487 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:16.168600 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":294,"threads_started":1}
I20260926 03:42:16.658905 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:16.657874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:51483)
0926 03:42:16.658549 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:16.658685 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":355,"threads_started":1}
I20260926 03:42:16.704803 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:16.703739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:57231)
0926 03:42:16.704427 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:16.704527 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":285,"threads_started":1}
I20260926 03:42:17.171267 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:17.170041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:49621)
0926 03:42:17.170845 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:17.170972 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":316,"threads_started":1}
I20260926 03:42:17.661818 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:17.660278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:49149)
0926 03:42:17.661141 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:17.661290 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":435,"threads_started":1}
I20260926 03:42:17.707224 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:17.706157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:60531)
0926 03:42:17.706847 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:17.706989 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":299,"threads_started":1}
I20260926 03:42:18.173723 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:18.172690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:34449)
0926 03:42:18.173393 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:18.173516 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":360,"threads_started":1}
I20260926 03:42:18.664252 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:18.663251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:48311)
0926 03:42:18.663912 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:18.664036 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":329,"threads_started":1}
I20260926 03:42:18.709398 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:18.708528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:43389)
0926 03:42:18.709105 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:18.709200 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
I20260926 03:42:19.176299 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:19.175204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:39339)
0926 03:42:19.175992 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:19.176106 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":382,"threads_started":1}
I20260926 03:42:19.666734 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:19.665530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:49437)
0926 03:42:19.666273 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:19.666500 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260926 03:42:19.711874 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:19.710868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:34611)
0926 03:42:19.711574 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:19.711688 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":339,"threads_started":1}
I20260926 03:42:20.178684 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:20.177594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:44065)
0926 03:42:20.178297 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:20.178444 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":367,"threads_started":1}
I20260926 03:42:20.669144 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:20.668257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:51843)
0926 03:42:20.668851 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:20.668971 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":313,"threads_started":1}
I20260926 03:42:20.714263 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:20.713221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:36789)
0926 03:42:20.713819 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:20.714002 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260926 03:42:21.181407 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:21.180161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:58081)
0926 03:42:21.180896 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:21.181067 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":329,"threads_started":1}
I20260926 03:42:21.671626 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:21.670420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:48567)
0926 03:42:21.671160 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:21.671305 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":380,"threads_started":1}
I20260926 03:42:21.716724 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:21.715669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:34625)
0926 03:42:21.716402 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:21.716532 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":348,"threads_started":1}
I20260926 03:42:22.183689 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:22.182514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:36091)
0926 03:42:22.183233 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:22.183413 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260926 03:42:22.673975 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:22.672993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:43091)
0926 03:42:22.673614 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:22.673748 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":304,"threads_started":1}
I20260926 03:42:22.718994 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:22.717995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:60745)
0926 03:42:22.718668 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:22.718793 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
I20260926 03:42:23.186244 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:23.185036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:33821)
0926 03:42:23.185804 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:23.185960 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":375,"threads_started":1}
I20260926 03:42:23.676131 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:23.675174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:46167)
0926 03:42:23.675796 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:23.675927 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":333,"threads_started":1}
I20260926 03:42:23.721264 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:23.720450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:32823)
0926 03:42:23.720965 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:23.721075 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":271,"threads_started":1}
I20260926 03:42:24.188060 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:24.187248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:45149)
0926 03:42:24.187776 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:24.187871 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260926 03:42:24.678318 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:24.677390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:48105)
0926 03:42:24.678024 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:24.678131 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":330,"threads_started":1}
I20260926 03:42:24.723644 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:24.722610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:41237)
0926 03:42:24.723271 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:24.723426 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":333,"threads_started":1}
I20260926 03:42:25.190523 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.189577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.130:52129)
0926 03:42:25.190197 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:25.190316 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":293,"threads_started":1}
I20260926 03:42:25.651556 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18570
I20260926 03:42:25.674343 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.653031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.1.221.129:41763)
0926 03:42:25.653785 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:25.653805 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:25.653852 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:42:25.655736 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:25.655749 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:25.656395 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:25.656844 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:25.658725 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.660253 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.661740 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.662592 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.662818 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:25.666486 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:25.666498 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:25.666524 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:25.666589 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:25.670105 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.671044 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.671053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.671063 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.671306 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.671767 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.671771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.671773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.672520 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:25.672862 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:25.673412 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 03:42:25.673994 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260926 03:42:25.676107 18733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:25.681661 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.679924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.131:38677)
0926 03:42:25.680694 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:25.680856 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":371,"threads_started":1}
I20260926 03:42:25.775365 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18570
I20260926 03:42:25.775637 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.773943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40107 (local address 127.1.221.129:55571)
0926 03:42:25.774698 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:25.774955 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":406,"threads_started":1}
I20260926 03:42:25.791052 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18741
I20260926 03:42:25.812323 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.792332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.1.221.130:45101)
0926 03:42:25.793068 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:25.793084 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:25.793131 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:42:25.795231 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:25.795242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:25.795801 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:25.796182 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:25.797779 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.799062 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.800373 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.801163 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.801347 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:25.804686 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:25.804697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:25.804717 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:25.804773 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:25.808310 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.809246 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.809255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.809262 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.809460 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.809865 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.809870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.809872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.810532 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:25.810841 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:25.811350 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 03:42:25.812012 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":369,"threads_started":1}
I20260926 03:42:25.814499 18864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:25.903725 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18741
I20260926 03:42:25.919951 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 18872
I20260926 03:42:25.939857 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:25.921256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.1.221.131:44329)
0926 03:42:25.922031 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:25.922046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:25.922089 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:42:25.923854 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:25.923865 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:25.924326 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:25.924688 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:25.926209 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.927486 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.928765 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:25.929450 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:25.929638 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:25.932488 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:25.932497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:25.932522 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:25.932566 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:25.935914 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.936708 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.936716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.936724 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.937049 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:25.937609 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:25.937614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:25.937616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:25.938254 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:25.938536 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:25.939052 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 03:42:25.939545 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
I20260926 03:42:25.941918 18995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:26.038779 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 18872
2026-09-26T03:42:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65605 ms)
[----------] 30 tests from SecurityITest (297412 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:26 dist-test-slave-lqrk krb5kdc[19058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:26 dist-test-slave-lqrk krb5kdc[19058](info): set up 2 sockets
Sep 26 03:42:26 dist-test-slave-lqrk krb5kdc[19058](info): commencing operation
krb5kdc: starting...
W20260926 03:42:27.095733 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:27 dist-test-slave-lqrk krb5kdc[19058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:42:27.351904 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38259
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:27.695542 19071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:27.696089 19071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:27.696359 19071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:27.704653 19071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:27.704795 19071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:27.704844 19071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:27.704913 19071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:27.704985 19071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:27.715327 19071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:27.716758 19071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:27.718077 19071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:27.733250 19076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:27.733186 19077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:27.733248 19079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:27.734673 19071 server_base.cc:1034] running on GCE node
I20260926 03:42:27.735672 19071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:27.737329 19071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:27.738631 19071 hybrid_clock.cc:648] HybridClock initialized: now 1790394147738533 us; error 82 us; skew 500 ppm
I20260926 03:42:27.739102 19071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:27 dist-test-slave-lqrk krb5kdc[19058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:27.743644 19071 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:42:27.745657 19071 webserver.cc:460] Webserver started at http://127.1.221.190:42837/ using document root <none> and password file <none>
I20260926 03:42:27.746446 19071 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:27.746582 19071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:27.746901 19071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:27.750115 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d"
format_stamp: "Formatted at 2026-09-26 03:42:27 on dist-test-slave-lqrk"
server_key: "2f2db19e685b0cbd013c3ed941c6b513"
server_key_iv: "9a7bd614a69e4de2f7dead65183de279"
server_key_version: "encryptionkey@0"
I20260926 03:42:27.750901 19071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d"
format_stamp: "Formatted at 2026-09-26 03:42:27 on dist-test-slave-lqrk"
server_key: "2f2db19e685b0cbd013c3ed941c6b513"
server_key_iv: "9a7bd614a69e4de2f7dead65183de279"
server_key_version: "encryptionkey@0"
I20260926 03:42:27.756910 19071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:42:27.761176 19086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:27.763316 19071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:42:27.763644 19071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d"
format_stamp: "Formatted at 2026-09-26 03:42:27 on dist-test-slave-lqrk"
server_key: "2f2db19e685b0cbd013c3ed941c6b513"
server_key_iv: "9a7bd614a69e4de2f7dead65183de279"
server_key_version: "encryptionkey@0"
I20260926 03:42:27.763859 19071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:27.782543 19071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:27.787364 19071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:27.787738 19071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:27.808557 19071 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:38259
I20260926 03:42:27.808642 19137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:38259 every 8 connection(s)
I20260926 03:42:27.810081 19071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:27.815296 19138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:27.819267 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19071
I20260926 03:42:27.819545 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:27.820181 1910 external_mini_cluster.cc:1428] Setting key 05079bb4427126972b1614f36bec9f39
I20260926 03:42:27.830310 19138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: Bootstrap starting.
I20260926 03:42:27.834355 19138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: Neither blocks nor log segments found. Creating new log.
Sep 26 03:42:27 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:27.835893 19138 log.cc:824] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:27.840378 19138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: No bootstrap required, opened a new log
I20260926 03:42:27.847309 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:27.822591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.1.221.190:38259)
0926 03:42:27.823647 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:27.823663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:27.823721 (+ 58us) server_negotiation.cc:407] Connection header received
0926 03:42:27.825775 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:27.825805 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:27.826344 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:27.826763 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:27.828535 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:27.829960 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:27.831483 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:27.832256 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:27.832444 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:27.837538 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:27.837553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:27.837580 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:27.837633 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:27.841237 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:27.842522 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:27.842530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:27.842539 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:27.842891 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:27.843515 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:27.843520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:27.843522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:27.844367 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:27.844664 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:27.845368 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 03:42:27.845818 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":414,"threads_started":1}
I20260926 03:42:27.851547 19138 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } }
I20260926 03:42:27.852082 19138 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:27.852188 19138 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1ad4cafef44f5ab68ca6f332f96b7d, State: Initialized, Role: FOLLOWER
I20260926 03:42:27.852927 19138 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } }
I20260926 03:42:27.853215 19138 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:27.853389 19138 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:27.853569 19138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:27.856259 19138 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } }
I20260926 03:42:27.856932 19138 leader_election.cc:302] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1ad4cafef44f5ab68ca6f332f96b7d; no voters:
I20260926 03:42:27.857700 19138 leader_election.cc:288] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:27.857986 19143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:27.859030 19143 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [term 1 LEADER]: Becoming Leader. State: Replica: fe1ad4cafef44f5ab68ca6f332f96b7d, State: Running, Role: LEADER
I20260926 03:42:27.859792 19143 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } }
I20260926 03:42:27.860453 19138 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:27.863620 19144 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } } }
I20260926 03:42:27.864713 19144 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:27.864754 19145 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1ad4cafef44f5ab68ca6f332f96b7d. Latest consensus state: current_term: 1 leader_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ad4cafef44f5ab68ca6f332f96b7d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38259 } } }
I20260926 03:42:27.865530 19145 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:27.868283 19149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:27.875030 19149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:27.886188 19149 catalog_manager.cc:1347] Generated new cluster ID: 541a87d396f24c20a157c1746a8b33ea
I20260926 03:42:27.886323 19149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:27.906543 19149 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:27.908114 19149 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:27.920521 19149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: Generated new TSK 0
I20260926 03:42:27.921504 19149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:42:27.993299 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:28.358008 19164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:28.358337 19164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:28.358486 19164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:28.366626 19164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:28.366783 19164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:28.366953 19164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:28.377974 19164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:28.379316 19164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:28.380676 19164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:28.395118 19172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:28.395179 19169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:28.395196 19170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:28.401585 19164 server_base.cc:1034] running on GCE node
I20260926 03:42:28.402403 19164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:28.403602 19164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:28.404871 19164 hybrid_clock.cc:648] HybridClock initialized: now 1790394148404807 us; error 77 us; skew 500 ppm
I20260926 03:42:28.405298 19164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:28 dist-test-slave-lqrk krb5kdc[19058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:28.409489 19164 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:42:28.411604 19164 webserver.cc:460] Webserver started at http://127.1.221.129:46759/ using document root <none> and password file <none>
I20260926 03:42:28.412402 19164 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:28.412525 19164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:28.412854 19164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:28.416014 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "1227e674fec1419e9a342e9bf45470fc"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "2d2308cc44bdd5dc1db785c2516b1f19"
server_key_iv: "b31d54ab56d43c9294286ac5eddeba9a"
server_key_version: "encryptionkey@0"
I20260926 03:42:28.416915 19164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "1227e674fec1419e9a342e9bf45470fc"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "2d2308cc44bdd5dc1db785c2516b1f19"
server_key_iv: "b31d54ab56d43c9294286ac5eddeba9a"
server_key_version: "encryptionkey@0"
I20260926 03:42:28.423514 19164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:42:28.429738 19179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:28.432356 19164 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260926 03:42:28.432636 19164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "1227e674fec1419e9a342e9bf45470fc"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "2d2308cc44bdd5dc1db785c2516b1f19"
server_key_iv: "b31d54ab56d43c9294286ac5eddeba9a"
server_key_version: "encryptionkey@0"
I20260926 03:42:28.432947 19164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:28.449122 19164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:28.454900 19164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:28.455279 19164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:28.456645 19164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:28.458492 19164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:28.458597 19164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:28.458717 19164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:28.458770 19164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:28.496755 19164 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41225
I20260926 03:42:28.496893 19291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41225 every 8 connection(s)
I20260926 03:42:28.498217 19164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:28.499508 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19164
I20260926 03:42:28.499740 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:28.500380 1910 external_mini_cluster.cc:1428] Setting key 070922e66e97fff6379dafe87b413533
Sep 26 03:42:28 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:28.523979 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:28.501623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44919 (local address 127.1.221.190:38259)
0926 03:42:28.502138 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:28.502143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:28.503926 (+ 1783us) server_negotiation.cc:407] Connection header received
0926 03:42:28.506272 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:28.506278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:28.506379 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:28.506533 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:28.508744 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:28.509767 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:28.512073 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:28.512536 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:28.512680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:28.517920 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:28.517926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:28.517930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:28.517971 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:28.520624 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:28.521524 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:28.521529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:28.521531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:28.521660 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:28.522095 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:28.522099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:28.522101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:28.522389 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:28.522522 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:28.523335 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 03:42:28.523625 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
I20260926 03:42:28.525112 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:28.502595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38259 (local address 127.1.221.129:44919)
0926 03:42:28.503673 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:28.503742 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:28.505884 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:28.506859 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:28.506875 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:28.507694 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:28.508500 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:28.508525 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:28.510031 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:28.510037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:28.511825 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:28.511851 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:28.512882 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:28.512891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:28.513296 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:28.514268 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:28.514303 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:28.517599 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:28.520844 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:28.520854 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:28.520870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:28.521386 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:28.521769 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:28.521774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:28.521777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:28.521974 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:28.522522 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:28.522531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:28.522918 (+ 387us) client_negotiation.cc:769] Sending connection context
0926 03:42:28.523186 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 03:42:28.523841 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":397,"threads_started":1}
I20260926 03:42:28.527295 19292 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38259
I20260926 03:42:28.527913 19292 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:42:28.529209 19292 heartbeater.cc:511] Master 127.1.221.190:38259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260926 03:42:28.532423 19103 ts_manager.cc:194] Registered new tserver with Master: 1227e674fec1419e9a342e9bf45470fc (127.1.221.129:41225)
I20260926 03:42:28.534299 19103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:44919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:42:28.567864 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:28.932806 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:28.933148 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:28.933290 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:28.941478 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:28.941627 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:28.941762 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:28.952585 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:28.954101 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:28.955729 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:28.971683 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:28.971725 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:28.971760 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:28.972683 19298 server_base.cc:1034] running on GCE node
I20260926 03:42:28.973412 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:28.974615 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:28.975891 19298 hybrid_clock.cc:648] HybridClock initialized: now 1790394148975808 us; error 75 us; skew 500 ppm
I20260926 03:42:28.976289 19298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:28 dist-test-slave-lqrk krb5kdc[19058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:28.980125 19298 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:42:28.982000 19298 webserver.cc:460] Webserver started at http://127.1.221.130:38375/ using document root <none> and password file <none>
I20260926 03:42:28.982772 19298 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:28.983158 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:28.984006 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:28.987521 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "f2e2c61b3597493ab826f8f67dc18554"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "a5c4918f70384eb224dfe47e9050d71a"
server_key_iv: "d6c968e7b65a4630831d72fd9ad24c79"
server_key_version: "encryptionkey@0"
I20260926 03:42:28.988464 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "f2e2c61b3597493ab826f8f67dc18554"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "a5c4918f70384eb224dfe47e9050d71a"
server_key_iv: "d6c968e7b65a4630831d72fd9ad24c79"
server_key_version: "encryptionkey@0"
I20260926 03:42:28.994848 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:42:28.999265 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.001071 19298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:42:29.001283 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "f2e2c61b3597493ab826f8f67dc18554"
format_stamp: "Formatted at 2026-09-26 03:42:28 on dist-test-slave-lqrk"
server_key: "a5c4918f70384eb224dfe47e9050d71a"
server_key_iv: "d6c968e7b65a4630831d72fd9ad24c79"
server_key_version: "encryptionkey@0"
I20260926 03:42:29.001518 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:29.017169 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:29.022384 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:29.022722 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:29.023958 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:29.026247 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:29.026404 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.026557 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:29.026628 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.066401 19298 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:43275
I20260926 03:42:29.066428 19425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:43275 every 8 connection(s)
I20260926 03:42:29.067859 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:29.078235 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19298
I20260926 03:42:29.078439 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:29.078934 1910 external_mini_cluster.cc:1428] Setting key 8feebba55a1264980ef5ce54ba7afd30
Sep 26 03:42:29 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:29.092819 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.071014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:33901 (local address 127.1.221.190:38259)
0926 03:42:29.071526 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.071533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.072665 (+ 1132us) server_negotiation.cc:407] Connection header received
0926 03:42:29.075213 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.075220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.075330 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.075587 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:29.077480 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.078489 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.080621 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.081303 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.081449 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.085607 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:29.085614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:29.085618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:29.085666 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:29.088967 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.090040 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.090046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.090050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.090216 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.090702 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.090708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.090711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.091090 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:29.091249 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.092159 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.092441 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":237,"threads_started":1}
I20260926 03:42:29.093698 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.071535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38259 (local address 127.1.221.130:33901)
0926 03:42:29.072323 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.072472 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.074576 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.075901 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.075913 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.076539 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:29.077225 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.077248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.078696 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.078700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.080420 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.080441 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.081577 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.081583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.081819 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.082563 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:29.082589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:29.085265 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:29.089221 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:29.089231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:29.089249 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:29.089815 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:29.090341 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:29.090345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:29.090347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:29.090559 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:29.091248 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:29.091257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:29.091706 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.092019 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.092481 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":298,"threads_started":1}
I20260926 03:42:29.095461 19426 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38259
I20260926 03:42:29.095909 19426 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:29.097043 19426 heartbeater.cc:511] Master 127.1.221.190:38259 requested a full tablet report, sending...
I20260926 03:42:29.099545 19103 ts_manager.cc:194] Registered new tserver with Master: f2e2c61b3597493ab826f8f67dc18554 (127.1.221.130:43275)
I20260926 03:42:29.100718 19103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:33901
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:42:29.144101 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:29.493307 19432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:29.493613 19432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:29.493731 19432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:29.504685 19432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:29.504870 19432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:29.505041 19432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:29.515960 19432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:29.517344 19432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:29.518667 19432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:29.533545 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:29.534432 19437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:29.534597 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:29.534999 19432 server_base.cc:1034] running on GCE node
I20260926 03:42:29.535856 19432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:29.536968 19432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:29.538194 19432 hybrid_clock.cc:648] HybridClock initialized: now 1790394149538132 us; error 59 us; skew 500 ppm
I20260926 03:42:29.538525 19292 heartbeater.cc:503] Master 127.1.221.190:38259 was elected leader, sending a full tablet report...
I20260926 03:42:29.538657 19432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:29 dist-test-slave-lqrk krb5kdc[19058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:29.542730 19432 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:42:29.547770 19432 webserver.cc:460] Webserver started at http://127.1.221.131:45103/ using document root <none> and password file <none>
I20260926 03:42:29.548727 19432 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:29.548841 19432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:29.549161 19432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:29.552413 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "5a574fd98e794d5686838026a8af4038"
format_stamp: "Formatted at 2026-09-26 03:42:29 on dist-test-slave-lqrk"
server_key: "db59e698d4e5e1e6e91a82b3775b53f3"
server_key_iv: "131100313c4ec5012d1d76a1411a30ba"
server_key_version: "encryptionkey@0"
I20260926 03:42:29.553236 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "5a574fd98e794d5686838026a8af4038"
format_stamp: "Formatted at 2026-09-26 03:42:29 on dist-test-slave-lqrk"
server_key: "db59e698d4e5e1e6e91a82b3775b53f3"
server_key_iv: "131100313c4ec5012d1d76a1411a30ba"
server_key_version: "encryptionkey@0"
I20260926 03:42:29.558702 19432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:42:29.562933 19447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.564493 19432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:29.564674 19432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "5a574fd98e794d5686838026a8af4038"
format_stamp: "Formatted at 2026-09-26 03:42:29 on dist-test-slave-lqrk"
server_key: "db59e698d4e5e1e6e91a82b3775b53f3"
server_key_iv: "131100313c4ec5012d1d76a1411a30ba"
server_key_version: "encryptionkey@0"
I20260926 03:42:29.564888 19432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:29.596604 19432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:29.601197 19432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:29.601475 19432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:29.602501 19432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:29.604058 19432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:29.604141 19432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.604266 19432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:29.604318 19432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:29.637884 19432 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44037
I20260926 03:42:29.637907 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44037 every 8 connection(s)
I20260926 03:42:29.639637 19432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:29.649951 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19432
I20260926 03:42:29.650149 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:29.650722 1910 external_mini_cluster.cc:1428] Setting key f173ccb2fecfcbccc330a8995d7179d9
Sep 26 03:42:29 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:29.663605 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.642701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35611 (local address 127.1.221.190:38259)
0926 03:42:29.643181 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.643186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.644150 (+ 964us) server_negotiation.cc:407] Connection header received
0926 03:42:29.646161 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.646167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.646273 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.646369 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:29.648144 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.648996 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.651512 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.652228 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.652394 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.656915 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:29.656922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:29.656925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:29.657006 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:29.659984 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.660988 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.660993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.660996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.661132 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.661573 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.661576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.661578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.661931 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:29.662088 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.662996 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.663209 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260926 03:42:29.664593 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.643136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38259 (local address 127.1.221.131:35611)
0926 03:42:29.643959 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.644006 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.645791 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.646655 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.646665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.647252 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:29.647935 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.647961 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.649188 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.649192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.651135 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.651155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.652791 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.652820 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.653273 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.654086 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:29.654112 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:29.656625 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:29.660197 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:29.660206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:29.660222 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:29.660792 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:29.661241 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:29.661245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:29.661247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:29.661474 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:29.662173 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:29.662181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:29.662596 (+ 415us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.662850 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.663330 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260926 03:42:29.666172 19560 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38259
I20260926 03:42:29.666486 19560 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:29.667235 19560 heartbeater.cc:511] Master 127.1.221.190:38259 requested a full tablet report, sending...
I20260926 03:42:29.668941 19102 ts_manager.cc:194] Registered new tserver with Master: 5a574fd98e794d5686838026a8af4038 (127.1.221.131:44037)
I20260926 03:42:29.669936 19102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:35611
I20260926 03:42:29.680410 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:29.699277 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.685548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.1.221.190:38259)
0926 03:42:29.685732 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.685738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.686238 (+ 500us) server_negotiation.cc:407] Connection header received
0926 03:42:29.686418 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.686421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.686493 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.686579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:29.688103 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.688842 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.690235 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.690667 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.690787 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.693422 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:29.693427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:29.693431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:29.693469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:29.696300 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.697122 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.697128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.697130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.697267 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:29.697664 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:29.697668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:29.697671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:29.698046 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:29.698304 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.698681 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.698933 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:42:29.705816 19102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:29.714836 19102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:29.758189 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.731989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.1.221.131:44037)
0926 03:42:29.748035 (+ 16046us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.748046 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.748973 (+ 927us) server_negotiation.cc:407] Connection header received
0926 03:42:29.749447 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.749456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.749699 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.749870 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:29.750993 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.753002 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.754933 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.756872 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757021 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757270 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.757608 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.757802 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15783,"thread_start_us":16066,"threads_started":1}
I20260926 03:42:29.758267 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.731659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44037 (local address 127.0.0.1:51438)
0926 03:42:29.748789 (+ 17130us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.748851 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.749242 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.750214 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.750217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.750242 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:29.750789 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.750803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.753144 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.753150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.754762 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.754778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.757089 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.757310 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757362 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.757540 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.757755 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16523,"mutex_wait_us":5,"thread_start_us":15988,"threads_started":1}
I20260926 03:42:29.758765 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.735849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.1.221.130:43275)
0926 03:42:29.740942 (+ 5093us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.740953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.749306 (+ 8353us) server_negotiation.cc:407] Connection header received
0926 03:42:29.749446 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.749464 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.749705 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.749890 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:29.750814 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.752201 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.755277 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.756872 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757020 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757236 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.758285 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.758482 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4845,"thread_start_us":4705,"threads_started":1}
I20260926 03:42:29.758834 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.732302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.1.221.129:41225)
0926 03:42:29.744382 (+ 12080us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.744392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.748981 (+ 4589us) server_negotiation.cc:407] Connection header received
0926 03:42:29.749454 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.749462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.749677 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.749807 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:29.750940 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.752561 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.755738 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.756944 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757096 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757327 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.758279 (+ 952us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.758514 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11869,"thread_start_us":11644,"threads_started":1}
I20260926 03:42:29.759117 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.731957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41225 (local address 127.0.0.1:56806)
0926 03:42:29.748825 (+ 16868us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.748858 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.749176 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.750112 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.750116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.750141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:29.750669 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.750678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.752677 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.752683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.755533 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.755545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.757482 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.757667 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757698 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.757772 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.757892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16735,"mutex_wait_us":69,"thread_start_us":17047,"threads_started":1}
I20260926 03:42:29.759208 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.736195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:43275 (local address 127.0.0.1:42580)
0926 03:42:29.749157 (+ 12962us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.749174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.749286 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.749957 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.749964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.749995 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:29.750589 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.750606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.752368 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.752373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.755097 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.755111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.757407 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.757412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.757582 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.757615 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.757703 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.757835 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12833,"mutex_wait_us":118,"thread_start_us":12336,"threads_started":1}
I20260926 03:42:29.762831 19227 tablet_service.cc:1467] Processing CreateTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 (DEFAULT_TABLE table=test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:29.762872 19495 tablet_service.cc:1467] Processing CreateTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 (DEFAULT_TABLE table=test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:29.764330 19227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bd29ffa56d47f0860a47a6fa88c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:29.764374 19495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bd29ffa56d47f0860a47a6fa88c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:29.764232 19361 tablet_service.cc:1467] Processing CreateTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 (DEFAULT_TABLE table=test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:29.765758 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bd29ffa56d47f0860a47a6fa88c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:29.777868 19579 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Bootstrap starting.
I20260926 03:42:29.782135 19579 tablet_bootstrap.cc:654] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:29.783314 19579 log.cc:824] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:29.785359 19580 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Bootstrap starting.
I20260926 03:42:29.786348 19579 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: No bootstrap required, opened a new log
I20260926 03:42:29.786608 19579 ts_tablet_manager.cc:1397] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:42:29.789441 19580 tablet_bootstrap.cc:654] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:29.790575 19580 log.cc:824] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:29.794710 19580 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: No bootstrap required, opened a new log
I20260926 03:42:29.795073 19580 ts_tablet_manager.cc:1397] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:42:29.795097 19581 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Bootstrap starting.
I20260926 03:42:29.795011 19579 raft_consensus.cc:348] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.795526 19579 raft_consensus.cc:374] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:29.795629 19579 raft_consensus.cc:729] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1227e674fec1419e9a342e9bf45470fc, State: Initialized, Role: FOLLOWER
I20260926 03:42:29.796277 19579 consensus_queue.cc:260] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.799178 19579 ts_tablet_manager.cc:1428] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 03:42:29.799768 19581 tablet_bootstrap.cc:654] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:29.801139 19581 log.cc:824] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:29.805058 19581 tablet_bootstrap.cc:492] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: No bootstrap required, opened a new log
I20260926 03:42:29.805395 19581 ts_tablet_manager.cc:1397] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20260926 03:42:29.805907 19580 raft_consensus.cc:348] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.806291 19580 raft_consensus.cc:374] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:29.806381 19580 raft_consensus.cc:729] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a574fd98e794d5686838026a8af4038, State: Initialized, Role: FOLLOWER
I20260926 03:42:29.807004 19580 consensus_queue.cc:260] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.808188 19560 heartbeater.cc:503] Master 127.1.221.190:38259 was elected leader, sending a full tablet report...
I20260926 03:42:29.810429 19580 ts_tablet_manager.cc:1428] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 03:42:29.816634 19581 raft_consensus.cc:348] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.817186 19581 raft_consensus.cc:374] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:29.817281 19581 raft_consensus.cc:729] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e2c61b3597493ab826f8f67dc18554, State: Initialized, Role: FOLLOWER
I20260926 03:42:29.817994 19581 consensus_queue.cc:260] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.819152 19426 heartbeater.cc:503] Master 127.1.221.190:38259 was elected leader, sending a full tablet report...
I20260926 03:42:29.819870 19581 ts_tablet_manager.cc:1428] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260926 03:42:29.820079 19427 tablet.cc:2369] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:29.850414 19586 raft_consensus.cc:482] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:29.850723 19586 raft_consensus.cc:504] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.852419 19586 leader_election.cc:288] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1227e674fec1419e9a342e9bf45470fc (127.1.221.129:41225), f2e2c61b3597493ab826f8f67dc18554 (127.1.221.130:43275)
I20260926 03:42:29.858129 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.852859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46175 (local address 127.1.221.129:41225)
0926 03:42:29.853160 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.853165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.853184 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:42:29.853426 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.853431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.853521 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.853629 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:29.854058 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.854799 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.856443 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.857252 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.857404 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.857482 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.857745 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.857874 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260926 03:42:29.858143 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.852806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41225 (local address 127.1.221.131:46175)
0926 03:42:29.853079 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.853098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.853240 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.853665 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.853673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.853705 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:29.853905 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.853916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.854938 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.854944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.856278 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.856292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.857404 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.857410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.857594 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.857620 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.857702 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.857844 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 03:42:29.858959 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.852895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46937 (local address 127.1.221.130:43275)
0926 03:42:29.853095 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.853101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.853775 (+ 674us) server_negotiation.cc:407] Connection header received
0926 03:42:29.853987 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.853992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.854056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.854158 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:29.854706 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.855444 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.857123 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.857979 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.858120 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.858191 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.858674 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.858780 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:42:29.859098 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.852972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:43275 (local address 127.1.221.131:46937)
0926 03:42:29.853684 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:29.853706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:29.853817 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:29.854205 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:29.854210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:29.854235 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:29.854538 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.854547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.855597 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.855602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.856957 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:29.856969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.858192 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.858341 (+ 149us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:29.858515 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.858547 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:42:29.858632 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:42:29.858759 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":370,"threads_started":1}
I20260926 03:42:29.859658 19247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bd29ffa56d47f0860a47a6fa88c028" candidate_uuid: "5a574fd98e794d5686838026a8af4038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1227e674fec1419e9a342e9bf45470fc" is_pre_election: true
I20260926 03:42:29.859892 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bd29ffa56d47f0860a47a6fa88c028" candidate_uuid: "5a574fd98e794d5686838026a8af4038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e2c61b3597493ab826f8f67dc18554" is_pre_election: true
I20260926 03:42:29.860177 19247 raft_consensus.cc:2454] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a574fd98e794d5686838026a8af4038 in term 0.
I20260926 03:42:29.860385 19381 raft_consensus.cc:2454] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a574fd98e794d5686838026a8af4038 in term 0.
I20260926 03:42:29.860916 19451 leader_election.cc:302] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1227e674fec1419e9a342e9bf45470fc, 5a574fd98e794d5686838026a8af4038; no voters:
I20260926 03:42:29.861388 19586 raft_consensus.cc:2781] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:29.861521 19586 raft_consensus.cc:482] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:42:29.861615 19586 raft_consensus.cc:3037] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:29.864127 19586 raft_consensus.cc:504] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.864781 19586 leader_election.cc:288] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [CANDIDATE]: Term 1 election: Requested vote from peers 1227e674fec1419e9a342e9bf45470fc (127.1.221.129:41225), f2e2c61b3597493ab826f8f67dc18554 (127.1.221.130:43275)
I20260926 03:42:29.865406 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bd29ffa56d47f0860a47a6fa88c028" candidate_uuid: "5a574fd98e794d5686838026a8af4038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e2c61b3597493ab826f8f67dc18554"
I20260926 03:42:29.865468 19247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bd29ffa56d47f0860a47a6fa88c028" candidate_uuid: "5a574fd98e794d5686838026a8af4038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1227e674fec1419e9a342e9bf45470fc"
I20260926 03:42:29.865609 19381 raft_consensus.cc:3037] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:29.865675 19247 raft_consensus.cc:3037] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:29.867996 19381 raft_consensus.cc:2454] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a574fd98e794d5686838026a8af4038 in term 1.
I20260926 03:42:29.868150 19247 raft_consensus.cc:2454] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a574fd98e794d5686838026a8af4038 in term 1.
I20260926 03:42:29.868541 19449 leader_election.cc:302] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a574fd98e794d5686838026a8af4038, f2e2c61b3597493ab826f8f67dc18554; no voters:
I20260926 03:42:29.868937 19586 raft_consensus.cc:2781] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:29.869423 19586 raft_consensus.cc:686] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 1 LEADER]: Becoming Leader. State: Replica: 5a574fd98e794d5686838026a8af4038, State: Running, Role: LEADER
I20260926 03:42:29.869974 19586 consensus_queue.cc:237] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } }
I20260926 03:42:29.874934 19102 catalog_manager.cc:5495] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a574fd98e794d5686838026a8af4038 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "5a574fd98e794d5686838026a8af4038" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a574fd98e794d5686838026a8af4038" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 } health_report { overall_health: UNKNOWN } } }
W20260926 03:42:29.891623 19561 tablet.cc:2369] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:29.917991 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.912532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.1.221.131:44037)
0926 03:42:29.912762 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.912766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.912782 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:42:29.912884 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.912889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.912969 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.913108 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:29.913515 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.914263 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.915796 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.916216 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.916328 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:29.916865 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:29.917399 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:29.917543 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:29.917642 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:42:29.917731 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:42:29.926687 19247 raft_consensus.cc:1261] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 1 FOLLOWER]: Refusing update from remote peer 5a574fd98e794d5686838026a8af4038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:29.926687 19381 raft_consensus.cc:1261] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 1 FOLLOWER]: Refusing update from remote peer 5a574fd98e794d5686838026a8af4038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:29.927809 19589 consensus_queue.cc:1035] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e2c61b3597493ab826f8f67dc18554" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:29.928176 19586 consensus_queue.cc:1035] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1227e674fec1419e9a342e9bf45470fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:29.957791 19102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:29.958173 19102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:29.962536 19102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe1ad4cafef44f5ab68ca6f332f96b7d: Sending DeleteTablet for 3 replicas of tablet 56bd29ffa56d47f0860a47a6fa88c028
I20260926 03:42:29.963982 19227 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:29 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56806
I20260926 03:42:29.964144 19495 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:29 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:51438
I20260926 03:42:29.964929 19602 tablet_replica.cc:330] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: stopping tablet replica
I20260926 03:42:29.965126 19361 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bd29ffa56d47f0860a47a6fa88c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:29 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:42580
I20260926 03:42:29.965505 19602 raft_consensus.cc:2229] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:29.966094 19602 raft_consensus.cc:2258] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:29.966343 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19164
I20260926 03:42:29.972492 19602 ts_tablet_manager.cc:1905] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:29.981549 19602 ts_tablet_manager.cc:1918] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:29.981761 19602 log.cc:1196] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/56bd29ffa56d47f0860a47a6fa88c028
I20260926 03:42:29.982316 19602 ts_tablet_manager.cc:1939] T 56bd29ffa56d47f0860a47a6fa88c028 P 1227e674fec1419e9a342e9bf45470fc: Deleting consensus metadata
I20260926 03:42:29.985111 19090 catalog_manager.cc:4841] TS 1227e674fec1419e9a342e9bf45470fc (127.1.221.129:41225): tablet 56bd29ffa56d47f0860a47a6fa88c028 (table test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]) successfully deleted
I20260926 03:42:29.986299 19604 tablet_replica.cc:330] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: stopping tablet replica
I20260926 03:42:29.986879 19604 raft_consensus.cc:2229] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:29.987608 19604 raft_consensus.cc:2258] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:29.989738 19603 tablet_replica.cc:330] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: stopping tablet replica
I20260926 03:42:29.990330 19603 raft_consensus.cc:2229] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:29.990577 19604 ts_tablet_manager.cc:1905] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:29.991976 19603 raft_consensus.cc:2258] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:29.995057 19603 ts_tablet_manager.cc:1905] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:42:29 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:42:30.002928 19604 ts_tablet_manager.cc:1918] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:30.003170 19604 log.cc:1196] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/56bd29ffa56d47f0860a47a6fa88c028
I20260926 03:42:30.003926 19604 ts_tablet_manager.cc:1939] T 56bd29ffa56d47f0860a47a6fa88c028 P f2e2c61b3597493ab826f8f67dc18554: Deleting consensus metadata
I20260926 03:42:30.006625 19603 ts_tablet_manager.cc:1918] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:30.006995 19603 log.cc:1196] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/56bd29ffa56d47f0860a47a6fa88c028
I20260926 03:42:30.006942 19088 catalog_manager.cc:4841] TS f2e2c61b3597493ab826f8f67dc18554 (127.1.221.130:43275): tablet 56bd29ffa56d47f0860a47a6fa88c028 (table test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]) successfully deleted
I20260926 03:42:30.007757 19603 ts_tablet_manager.cc:1939] T 56bd29ffa56d47f0860a47a6fa88c028 P 5a574fd98e794d5686838026a8af4038: Deleting consensus metadata
I20260926 03:42:30.009680 19089 catalog_manager.cc:4841] TS 5a574fd98e794d5686838026a8af4038 (127.1.221.131:44037): tablet 56bd29ffa56d47f0860a47a6fa88c028 (table test-table [id=b3b6161a70cc4f19b8d19cf3c0644088]) successfully deleted
I20260926 03:42:30.014472 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:29.968526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.1.221.129:41225)
0926 03:42:29.969235 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:29.969243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:29.989140 (+ 19897us) server_negotiation.cc:407] Connection header received
0926 03:42:29.989342 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:29.989347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:29.989445 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:29.989526 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:29.991610 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.992766 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.995020 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:29.995747 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:29.995883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:30.004676 (+ 8793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:30.004714 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:30.004746 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:30.004832 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:30.010112 (+ 5280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.011316 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.011330 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.011334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.011611 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.012162 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.012169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.012172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.012746 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:30.013300 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:30.013739 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 03:42:30.014067 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260926 03:42:30.015334 19287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:30.124181 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19164
I20260926 03:42:30.139916 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19298
Sep 26 03:42:30 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:42:30.159758 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:30.140847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.1.221.130:43275)
0926 03:42:30.141044 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:30.141051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:30.141073 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:42:30.141142 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:30.141148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:30.141229 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:30.141356 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:30.143186 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:30.144434 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:30.146227 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:30.146892 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:30.147052 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:30.151750 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:30.151773 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:30.151798 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:30.151853 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:30.155692 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.156598 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.156609 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.156613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.156779 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.157248 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.157253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.157257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.157773 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:30.158478 (+ 705us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:30.158837 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:42:30.159216 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:42:30.160748 19421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:30.297689 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19298
I20260926 03:42:30.313210 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19432
Sep 26 03:42:30 dist-test-slave-lqrk krb5kdc[19058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:42:30.334291 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:30.313989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.1.221.131:44037)
0926 03:42:30.314137 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:30.314142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:30.314187 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:42:30.314709 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:30.314735 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:30.314896 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:30.315021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:30.316959 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:30.317759 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:30.319813 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:30.320411 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:30.320618 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:30.326720 (+ 6102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:30.326746 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:30.326769 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:30.326828 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:30.330385 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.331200 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.331209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.331212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.331445 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:30.332077 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:30.332086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:30.332090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:30.332923 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:30.333232 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:30.333695 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 03:42:30.333947 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:42:30.335095 19555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:30.466487 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19432
I20260926 03:42:30.483050 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19071
I20260926 03:42:30.484321 19133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:30.600070 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19071
2026-09-26T03:42:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4558 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T03:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:30Z Disabled control of system clock
I20260926 03:42:30.678603 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33041
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:31.032081 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:31.032415 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:31.032545 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:31.040009 19613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:31.040151 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:31.040207 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:31.040287 19613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:31.040349 19613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:31.049937 19613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:31.051316 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:31.053259 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:31.067119 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:31.067248 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:31.067130 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:31.068178 19613 server_base.cc:1034] running on GCE node
I20260926 03:42:31.069428 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:31.071957 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:31.073428 19613 hybrid_clock.cc:648] HybridClock initialized: now 1790394151073343 us; error 58 us; skew 500 ppm
I20260926 03:42:31.074151 19613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:31.078722 19613 webserver.cc:460] Webserver started at http://127.1.221.190:42863/ using document root <none> and password file <none>
I20260926 03:42:31.080278 19613 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:31.080477 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:31.081097 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:31.086768 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "12bd0756e11b41ada151d1b00a25eb93"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "46205a09717cf8911ff9e917936275a2"
server_key_iv: "253c59caf5c80a2698f48770229e8861"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.088325 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "12bd0756e11b41ada151d1b00a25eb93"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "46205a09717cf8911ff9e917936275a2"
server_key_iv: "253c59caf5c80a2698f48770229e8861"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.098976 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260926 03:42:31.105909 19627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:31.108512 19613 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:42:31.108886 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "12bd0756e11b41ada151d1b00a25eb93"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "46205a09717cf8911ff9e917936275a2"
server_key_iv: "253c59caf5c80a2698f48770229e8861"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.109211 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:31.147090 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:31.148303 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:31.148635 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:31.170573 19613 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33041
I20260926 03:42:31.170596 19678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33041 every 8 connection(s)
I20260926 03:42:31.172401 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:31.176029 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19613
I20260926 03:42:31.176221 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:31.176800 1910 external_mini_cluster.cc:1428] Setting key 6c0a70235b56d2bb35d3c33db9485f88
I20260926 03:42:31.177601 19679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:31.196642 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:31.179263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.1.221.190:33041)
0926 03:42:31.180798 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:31.180826 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:31.180896 (+ 70us) server_negotiation.cc:407] Connection header received
0926 03:42:31.183082 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:31.183121 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:31.184103 (+ 982us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:31.184734 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:31.185475 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.187871 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.189554 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.190768 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.191025 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:31.192571 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:31.192586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:31.192625 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:31.193647 (+ 1022us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:31.193800 (+ 153us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:31.193838 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:31.194024 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:31.194366 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:42:31.194887 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":444,"threads_started":1}
I20260926 03:42:31.197392 19679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: Bootstrap starting.
I20260926 03:42:31.202133 19679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:31.203768 19679 log.cc:824] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:31.207881 19679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: No bootstrap required, opened a new log
I20260926 03:42:31.217314 19679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } }
I20260926 03:42:31.218047 19679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:31.218235 19679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bd0756e11b41ada151d1b00a25eb93, State: Initialized, Role: FOLLOWER
I20260926 03:42:31.219146 19679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } }
I20260926 03:42:31.219564 19679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:31.219730 19679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:31.219947 19679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:31.222638 19679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } }
I20260926 03:42:31.223480 19679 leader_election.cc:302] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12bd0756e11b41ada151d1b00a25eb93; no voters:
I20260926 03:42:31.224383 19679 leader_election.cc:288] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:31.224602 19684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:31.225629 19684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [term 1 LEADER]: Becoming Leader. State: Replica: 12bd0756e11b41ada151d1b00a25eb93, State: Running, Role: LEADER
I20260926 03:42:31.226297 19684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } }
I20260926 03:42:31.227617 19679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:31.229349 19686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12bd0756e11b41ada151d1b00a25eb93. Latest consensus state: current_term: 1 leader_uuid: "12bd0756e11b41ada151d1b00a25eb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } } }
I20260926 03:42:31.229921 19686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:31.229375 19685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12bd0756e11b41ada151d1b00a25eb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12bd0756e11b41ada151d1b00a25eb93" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33041 } } }
I20260926 03:42:31.230134 19685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:31.230906 19689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:31.239135 19689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:31.248963 19689 catalog_manager.cc:1347] Generated new cluster ID: 5454eb94518b45899f5531cff1292fa6
I20260926 03:42:31.249105 19689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:31.262281 19689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:31.263868 19689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:31.275271 19689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: Generated new TSK 0
I20260926 03:42:31.276356 19689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:31.285969 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--builtin_ntp_servers=127.1.221.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:31.625715 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:31.626026 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:31.626127 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:31.633772 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:31.633888 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:31.634029 19703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:31.643601 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:31.644915 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:31.646714 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:31.661149 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:31.661242 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:31.661211 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:31.662390 19703 server_base.cc:1034] running on GCE node
I20260926 03:42:31.663316 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:31.664692 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:31.665983 19703 hybrid_clock.cc:648] HybridClock initialized: now 1790394151665913 us; error 45 us; skew 500 ppm
I20260926 03:42:31.666476 19703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:31.669888 19703 webserver.cc:460] Webserver started at http://127.1.221.129:41447/ using document root <none> and password file <none>
I20260926 03:42:31.670948 19703 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:31.671097 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:31.671608 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:31.675597 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "19ac5e1dc98fda4279b95de4f2077edc"
server_key_iv: "a463a949c341712246282384e809f114"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.676666 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "19ac5e1dc98fda4279b95de4f2077edc"
server_key_iv: "a463a949c341712246282384e809f114"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.683863 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 03:42:31.687932 19717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:31.689538 19703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:42:31.689723 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85"
format_stamp: "Formatted at 2026-09-26 03:42:31 on dist-test-slave-lqrk"
server_key: "19ac5e1dc98fda4279b95de4f2077edc"
server_key_iv: "a463a949c341712246282384e809f114"
server_key_version: "encryptionkey@0"
I20260926 03:42:31.690007 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:31.708720 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:31.709600 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:31.709887 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:31.710914 19703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:31.712546 19703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:31.712637 19703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:31.712721 19703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:31.712775 19703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:31.744146 19703 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:35545
I20260926 03:42:31.744180 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:35545 every 8 connection(s)
I20260926 03:42:31.745602 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:31.753109 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19703
I20260926 03:42:31.753320 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:31.753769 1910 external_mini_cluster.cc:1428] Setting key 33867437e3a5f068539377ced82d54f6
I20260926 03:42:31.756670 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--builtin_ntp_servers=127.1.221.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:31.760121 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:31.748480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:48813 (local address 127.1.221.190:33041)
0926 03:42:31.749087 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:31.749094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:31.750499 (+ 1405us) server_negotiation.cc:407] Connection header received
0926 03:42:31.752697 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:31.752704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:31.752820 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:31.752963 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:31.754348 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.755197 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.757031 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.757488 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.757620 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:31.759201 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:31.759206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:31.759210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:31.759233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:31.759279 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:31.759293 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:31.759374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:31.759637 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:42:31.759762 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":226,"threads_started":1}
I20260926 03:42:31.761278 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:31.749221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33041 (local address 127.1.221.129:48813)
0926 03:42:31.750284 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:31.750360 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:31.752218 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:31.753275 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:31.753288 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:31.753432 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:31.754137 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:31.754158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.755371 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.755375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:31.756831 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:31.756855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:31.757772 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:31.757780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:31.758047 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:31.758782 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:31.758784 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:31.758833 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:31.758930 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:31.759416 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:31.759429 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:31.759437 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:42:31.759612 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 03:42:31.760105 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":326,"threads_started":1}
I20260926 03:42:31.763237 19830 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33041
I20260926 03:42:31.763623 19830 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:31.764704 19830 heartbeater.cc:511] Master 127.1.221.190:33041 requested a full tablet report, sending...
I20260926 03:42:31.767223 19644 ts_manager.cc:194] Registered new tserver with Master: c4cd9b34e6b840e8a6b2cd89aa096e85 (127.1.221.129:35545)
I20260926 03:42:31.768958 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:48813
W20260926 03:42:32.112668 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:32.112998 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:32.113121 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:32.120754 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:32.120890 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:32.121026 19834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:32.131991 19834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:32.133338 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:32.135569 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:32.150420 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:32.150519 19839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:32.150520 19840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:32.150976 19834 server_base.cc:1034] running on GCE node
I20260926 03:42:32.151898 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:32.153062 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:32.154332 19834 hybrid_clock.cc:648] HybridClock initialized: now 1790394152154273 us; error 53 us; skew 500 ppm
I20260926 03:42:32.154723 19834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:32.157248 19834 webserver.cc:460] Webserver started at http://127.1.221.130:44827/ using document root <none> and password file <none>
I20260926 03:42:32.158002 19834 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:32.158108 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:32.158435 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:32.161680 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "85f5085e0ceb40bf827818614546e614"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "e741cf6d5f1caac8748be25caba9b4d3"
server_key_iv: "2b1cdde6c39455d76fdb62cefc9fee64"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.163297 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "85f5085e0ceb40bf827818614546e614"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "e741cf6d5f1caac8748be25caba9b4d3"
server_key_iv: "2b1cdde6c39455d76fdb62cefc9fee64"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.169417 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:42:32.173346 19848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.175058 19834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:42:32.175239 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "85f5085e0ceb40bf827818614546e614"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "e741cf6d5f1caac8748be25caba9b4d3"
server_key_iv: "2b1cdde6c39455d76fdb62cefc9fee64"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.175521 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:32.195037 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:32.196038 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:32.196342 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:32.197397 19834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:32.199047 19834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:32.199138 19834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.199223 19834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:32.199290 19834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.234958 19834 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:35921
I20260926 03:42:32.234968 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:35921 every 8 connection(s)
I20260926 03:42:32.236490 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:32.244175 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19834
I20260926 03:42:32.244369 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:32.245088 1910 external_mini_cluster.cc:1428] Setting key cd6be547753680e25ea1c87681839ef9
I20260926 03:42:32.248275 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--builtin_ntp_servers=127.1.221.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:32.252033 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.239914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:44237 (local address 127.1.221.190:33041)
0926 03:42:32.240134 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.240140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.241689 (+ 1549us) server_negotiation.cc:407] Connection header received
0926 03:42:32.243609 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.243614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.243716 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.243802 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:32.245592 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.246620 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.248574 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.249188 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.249289 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.251106 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:32.251112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:32.251115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:32.251135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:32.251171 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:32.251184 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:32.251264 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.251557 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.251675 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:42:32.253299 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.240452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33041 (local address 127.1.221.130:44237)
0926 03:42:32.241503 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.241558 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.243238 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.244103 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.244117 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.244255 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:32.245334 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.245366 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.246812 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.246816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.248358 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.248376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.249591 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.249597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.249973 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.250721 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:32.250724 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:32.250771 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:32.250878 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:32.251281 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:32.251293 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:32.251301 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.251534 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.252056 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":317,"threads_started":1}
I20260926 03:42:32.255355 19961 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33041
I20260926 03:42:32.255939 19961 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:32.256799 19961 heartbeater.cc:511] Master 127.1.221.190:33041 requested a full tablet report, sending...
I20260926 03:42:32.258631 19644 ts_manager.cc:194] Registered new tserver with Master: 85f5085e0ceb40bf827818614546e614 (127.1.221.130:35921)
I20260926 03:42:32.259771 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:44237
W20260926 03:42:32.602155 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:32.602469 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:32.602555 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:32.610140 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:32.610258 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:32.610365 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:32.620548 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:32.621902 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:32.624495 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:32.639159 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:32.639184 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:32.639360 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:32.640143 19964 server_base.cc:1034] running on GCE node
I20260926 03:42:32.640817 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:32.641960 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:32.643205 19964 hybrid_clock.cc:648] HybridClock initialized: now 1790394152643147 us; error 48 us; skew 500 ppm
I20260926 03:42:32.643633 19964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:32.646320 19964 webserver.cc:460] Webserver started at http://127.1.221.131:41647/ using document root <none> and password file <none>
I20260926 03:42:32.647097 19964 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:32.647205 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:32.647570 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:32.650970 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "37dfca623011422199b11fb56f8631ad"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "daf334ef4b386338a1440051225e1325"
server_key_iv: "e4da110401447620d42a41055ee21c21"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.652782 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "37dfca623011422199b11fb56f8631ad"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "daf334ef4b386338a1440051225e1325"
server_key_iv: "e4da110401447620d42a41055ee21c21"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.659926 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 03:42:32.664178 19978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.665898 19964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:42:32.666097 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "37dfca623011422199b11fb56f8631ad"
format_stamp: "Formatted at 2026-09-26 03:42:32 on dist-test-slave-lqrk"
server_key: "daf334ef4b386338a1440051225e1325"
server_key_iv: "e4da110401447620d42a41055ee21c21"
server_key_version: "encryptionkey@0"
I20260926 03:42:32.666327 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:32.692325 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:32.693250 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:32.693614 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:32.694648 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:32.696722 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:32.697054 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.697203 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:32.697259 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:32.731978 19964 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36407
I20260926 03:42:32.732074 20090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36407 every 8 connection(s)
I20260926 03:42:32.733289 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:32.737133 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 19964
I20260926 03:42:32.737320 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:32.737856 1910 external_mini_cluster.cc:1428] Setting key f0d91ec5611249128b6e2a7b0874390f
I20260926 03:42:32.747792 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.736011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57199 (local address 127.1.221.190:33041)
0926 03:42:32.736179 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.736184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.737992 (+ 1808us) server_negotiation.cc:407] Connection header received
0926 03:42:32.740326 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.740333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.740433 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.740519 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:32.741800 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.742563 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.744472 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.744930 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.745034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.746862 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:32.746868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:32.746872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:32.746894 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:32.746934 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:32.746955 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:32.747043 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.747346 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.747495 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:42:32.749177 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.736645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33041 (local address 127.1.221.131:57199)
0926 03:42:32.737753 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.737801 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.739976 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.740812 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.740824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.740981 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:32.741613 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.741633 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.742745 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.742748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.744218 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.744234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.745184 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.745191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.745548 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.746370 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:32.746373 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:32.746430 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:32.746564 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:32.747066 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:32.747077 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:32.747086 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.747286 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.747776 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260926 03:42:32.751097 20091 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33041
I20260926 03:42:32.751528 20091 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:32.752511 20091 heartbeater.cc:511] Master 127.1.221.190:33041 requested a full tablet report, sending...
I20260926 03:42:32.754571 19644 ts_manager.cc:194] Registered new tserver with Master: 37dfca623011422199b11fb56f8631ad (127.1.221.131:36407)
I20260926 03:42:32.755551 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:57199
I20260926 03:42:32.768221 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:32.772466 19830 heartbeater.cc:503] Master 127.1.221.190:33041 was elected leader, sending a full tablet report...
I20260926 03:42:32.780409 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.772560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.1.221.190:33041)
0926 03:42:32.772718 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.772723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.773349 (+ 626us) server_negotiation.cc:407] Connection header received
0926 03:42:32.773502 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.773507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.773600 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.773714 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:32.774252 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.775063 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.777028 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.777752 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.777953 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.779433 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:32.779442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:32.779447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:32.779477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:32.779536 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:32.779555 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:32.779661 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.779957 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.780088 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:42:32.786087 19644 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:32.793315 19644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:32.827116 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.809118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:35545 (local address 127.0.0.1:48556)
0926 03:42:32.812382 (+ 3264us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.812454 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.812938 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.815713 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.815723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.815781 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:32.816975 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.816998 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.820083 (+ 3085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.820090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.823825 (+ 3735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.823860 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.826097 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.826105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.826325 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.826394 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.826529 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.826761 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2709,"thread_start_us":2733,"threads_started":1}
I20260926 03:42:32.827844 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.809839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.1.221.129:35545)
0926 03:42:32.811437 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.811451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.812751 (+ 1300us) server_negotiation.cc:407] Connection header received
0926 03:42:32.813450 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.813461 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.814102 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.815133 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:32.817365 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.819814 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.824243 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.825855 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.826026 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.826344 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.827253 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.827501 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1142,"threads_started":1}
I20260926 03:42:32.829000 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.809817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:35921 (local address 127.0.0.1:48140)
0926 03:42:32.813183 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.813206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.813493 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.815768 (+ 2275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.815772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.815791 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:32.816975 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.816998 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.820083 (+ 3085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.820090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.825607 (+ 5517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.825627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.828123 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.828131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.828335 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.828408 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.828499 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.828666 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3001,"mutex_wait_us":2323,"thread_start_us":2640,"threads_started":1}
I20260926 03:42:32.829078 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.810801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.1.221.130:35921)
0926 03:42:32.811790 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.811800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.813357 (+ 1557us) server_negotiation.cc:407] Connection header received
0926 03:42:32.814031 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.814041 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.814690 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.815159 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:32.817347 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.819814 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.825851 (+ 6037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.827566 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.827726 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.827986 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.828564 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.828734 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":499,"threads_started":1}
I20260926 03:42:32.832747 19765 tablet_service.cc:1467] Processing CreateTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e (DEFAULT_TABLE table=test-table [id=431deb284eba45fabd0fedeb02451e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:32.833014 19896 tablet_service.cc:1467] Processing CreateTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e (DEFAULT_TABLE table=test-table [id=431deb284eba45fabd0fedeb02451e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:32.834275 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.813500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36407 (local address 127.0.0.1:34620)
0926 03:42:32.824472 (+ 10972us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.824502 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.824675 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.826516 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.826520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.826545 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:32.826961 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.826982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.829244 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.829250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.831292 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.831350 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.833407 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.833414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.833606 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.833645 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.833758 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.833927 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10762,"thread_start_us":10376,"threads_started":1}
I20260926 03:42:32.834312 19765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450c3ae0bf1645ef898dc0f60c113b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:32.834395 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.814196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.1.221.131:36407)
0926 03:42:32.823166 (+ 8970us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.823182 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.824640 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 03:42:32.824864 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.824871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.825147 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.825356 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:32.827195 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.829112 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.831660 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.833152 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.833326 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.833619 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.833817 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.834003 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8599,"thread_start_us":8462,"threads_started":1}
I20260926 03:42:32.834412 19896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450c3ae0bf1645ef898dc0f60c113b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:32.838450 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e (DEFAULT_TABLE table=test-table [id=431deb284eba45fabd0fedeb02451e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:32.840852 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450c3ae0bf1645ef898dc0f60c113b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:32.848469 20109 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Bootstrap starting.
I20260926 03:42:32.852437 20110 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Bootstrap starting.
I20260926 03:42:32.853341 20109 tablet_bootstrap.cc:654] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:32.855746 20109 log.cc:824] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:32.857002 20110 tablet_bootstrap.cc:654] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:32.858691 20110 log.cc:824] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:32.859501 20112 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Bootstrap starting.
I20260926 03:42:32.860020 20109 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: No bootstrap required, opened a new log
I20260926 03:42:32.860392 20109 ts_tablet_manager.cc:1397] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 03:42:32.864719 20112 tablet_bootstrap.cc:654] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:32.866894 20112 log.cc:824] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:32.867784 20110 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: No bootstrap required, opened a new log
I20260926 03:42:32.868202 20110 ts_tablet_manager.cc:1397] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260926 03:42:32.871965 20109 raft_consensus.cc:348] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.872382 20109 raft_consensus.cc:374] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:32.872473 20109 raft_consensus.cc:729] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd9b34e6b840e8a6b2cd89aa096e85, State: Initialized, Role: FOLLOWER
I20260926 03:42:32.873113 20109 consensus_queue.cc:260] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.875882 20109 ts_tablet_manager.cc:1428] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 03:42:32.876911 20112 tablet_bootstrap.cc:492] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: No bootstrap required, opened a new log
I20260926 03:42:32.877241 20112 ts_tablet_manager.cc:1397] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260926 03:42:32.879055 20110 raft_consensus.cc:348] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.879606 20110 raft_consensus.cc:374] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:32.879714 20110 raft_consensus.cc:729] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f5085e0ceb40bf827818614546e614, State: Initialized, Role: FOLLOWER
I20260926 03:42:32.880537 20110 consensus_queue.cc:260] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.882709 19961 heartbeater.cc:503] Master 127.1.221.190:33041 was elected leader, sending a full tablet report...
I20260926 03:42:32.883483 20110 ts_tablet_manager.cc:1428] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 03:42:32.889031 20112 raft_consensus.cc:348] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.889536 20112 raft_consensus.cc:374] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:32.889647 20112 raft_consensus.cc:729] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfca623011422199b11fb56f8631ad, State: Initialized, Role: FOLLOWER
I20260926 03:42:32.890223 20112 consensus_queue.cc:260] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.891428 20091 heartbeater.cc:503] Master 127.1.221.190:33041 was elected leader, sending a full tablet report...
I20260926 03:42:32.891974 20112 ts_tablet_manager.cc:1428] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 03:42:32.976061 20116 raft_consensus.cc:482] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:32.976359 20116 raft_consensus.cc:504] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.978078 20116 leader_election.cc:288] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4cd9b34e6b840e8a6b2cd89aa096e85 (127.1.221.129:35545), 37dfca623011422199b11fb56f8631ad (127.1.221.131:36407)
I20260926 03:42:32.985117 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.978626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:38333 (local address 127.1.221.129:35545)
0926 03:42:32.978861 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.978865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.979215 (+ 350us) server_negotiation.cc:407] Connection header received
0926 03:42:32.979462 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.979468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.979559 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.979663 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:32.980380 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.981237 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.983228 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.984126 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.984252 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.984303 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.984714 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.984845 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 03:42:32.985061 20092 tablet.cc:2369] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:32.985193 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.979223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36407 (local address 127.1.221.130:35499)
0926 03:42:32.979881 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.979895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.980003 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.980310 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.980315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.980340 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:32.980661 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.980672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.981743 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.981746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.983214 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.983228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.984444 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.984448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.984595 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.984616 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.984713 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.984851 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":41,"thread_start_us":304,"threads_started":1}
I20260926 03:42:32.985642 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.978454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:35545 (local address 127.1.221.130:38333)
0926 03:42:32.979096 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:32.979115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:32.979266 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:32.979732 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:32.979736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:32.979762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:32.980219 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.980233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.981367 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.981373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.983025 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:32.983037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.984322 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.984329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:32.984553 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.984577 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:42:32.984649 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:42:32.984787 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":63,"thread_start_us":300,"threads_started":1}
I20260926 03:42:32.986207 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:32.979463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35499 (local address 127.1.221.131:36407)
0926 03:42:32.979638 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:32.979644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:32.979976 (+ 332us) server_negotiation.cc:407] Connection header received
0926 03:42:32.980114 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:32.980119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:32.980196 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:32.980288 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:32.980802 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.981608 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.983422 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:32.984256 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:32.984415 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:32.984482 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:32.985895 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0926 03:42:32.986017 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:42:32.986691 19785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450c3ae0bf1645ef898dc0f60c113b3e" candidate_uuid: "85f5085e0ceb40bf827818614546e614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" is_pre_election: true
I20260926 03:42:32.987097 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450c3ae0bf1645ef898dc0f60c113b3e" candidate_uuid: "85f5085e0ceb40bf827818614546e614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfca623011422199b11fb56f8631ad" is_pre_election: true
I20260926 03:42:32.987288 19785 raft_consensus.cc:2454] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f5085e0ceb40bf827818614546e614 in term 0.
I20260926 03:42:32.987619 20046 raft_consensus.cc:2454] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f5085e0ceb40bf827818614546e614 in term 0.
I20260926 03:42:32.988143 19851 leader_election.cc:302] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f5085e0ceb40bf827818614546e614, c4cd9b34e6b840e8a6b2cd89aa096e85; no voters:
I20260926 03:42:32.988564 20116 raft_consensus.cc:2781] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:32.988690 20116 raft_consensus.cc:482] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260926 03:42:32.988960 19962 tablet.cc:2369] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:32.989084 20116 raft_consensus.cc:3037] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:32.991600 20116 raft_consensus.cc:504] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:32.992168 20116 leader_election.cc:288] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [CANDIDATE]: Term 1 election: Requested vote from peers c4cd9b34e6b840e8a6b2cd89aa096e85 (127.1.221.129:35545), 37dfca623011422199b11fb56f8631ad (127.1.221.131:36407)
I20260926 03:42:32.992702 19785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450c3ae0bf1645ef898dc0f60c113b3e" candidate_uuid: "85f5085e0ceb40bf827818614546e614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85"
I20260926 03:42:32.992769 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450c3ae0bf1645ef898dc0f60c113b3e" candidate_uuid: "85f5085e0ceb40bf827818614546e614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfca623011422199b11fb56f8631ad"
I20260926 03:42:32.992959 19785 raft_consensus.cc:3037] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:32.992981 20046 raft_consensus.cc:3037] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:32.996253 19785 raft_consensus.cc:2454] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f5085e0ceb40bf827818614546e614 in term 1.
I20260926 03:42:32.996253 20046 raft_consensus.cc:2454] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f5085e0ceb40bf827818614546e614 in term 1.
I20260926 03:42:32.996794 19851 leader_election.cc:302] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f5085e0ceb40bf827818614546e614, c4cd9b34e6b840e8a6b2cd89aa096e85; no voters:
I20260926 03:42:32.997138 20116 raft_consensus.cc:2781] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:32.997689 20116 raft_consensus.cc:686] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 1 LEADER]: Becoming Leader. State: Replica: 85f5085e0ceb40bf827818614546e614, State: Running, Role: LEADER
W20260926 03:42:32.998145 19831 tablet.cc:2369] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:32.998356 20116 consensus_queue.cc:237] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } }
I20260926 03:42:33.004741 19643 catalog_manager.cc:5495] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f5085e0ceb40bf827818614546e614 (127.1.221.130). New cstate: current_term: 1 leader_uuid: "85f5085e0ceb40bf827818614546e614" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f5085e0ceb40bf827818614546e614" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 } health_report { overall_health: UNKNOWN } } }
I20260926 03:42:33.022459 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:33.017188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.1.221.130:35921)
0926 03:42:33.017401 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:33.017407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:33.017427 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:42:33.017528 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:33.017532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:33.017629 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:33.017763 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:33.018122 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.018840 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.020299 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.020714 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.020823 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:33.021338 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:33.021866 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:33.022007 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:33.022107 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:42:33.022188 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:42:33.031213 19785 raft_consensus.cc:1261] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 1 FOLLOWER]: Refusing update from remote peer 85f5085e0ceb40bf827818614546e614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:33.031213 20046 raft_consensus.cc:1261] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 1 FOLLOWER]: Refusing update from remote peer 85f5085e0ceb40bf827818614546e614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:33.032397 20120 consensus_queue.cc:1035] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cd9b34e6b840e8a6b2cd89aa096e85" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:33.032853 20116 consensus_queue.cc:1035] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfca623011422199b11fb56f8631ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:33.045423 20123 mvcc.cc:205] Tried to move back new op lower bound from 7333454450804801536 to 7333454450688778240. Current Snapshot: MvccSnapshot[applied={T|T < 7333454450804801536}]
I20260926 03:42:33.060922 19643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:33.061219 19643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:33.065671 19643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12bd0756e11b41ada151d1b00a25eb93: Sending DeleteTablet for 3 replicas of tablet 450c3ae0bf1645ef898dc0f60c113b3e
I20260926 03:42:33.067603 19765 tablet_service.cc:1514] Processing DeleteTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:33 UTC) from {username='slave'} at 127.0.0.1:48556
I20260926 03:42:33.067591 20026 tablet_service.cc:1514] Processing DeleteTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:33 UTC) from {username='slave'} at 127.0.0.1:34620
I20260926 03:42:33.068641 20133 tablet_replica.cc:330] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: stopping tablet replica
I20260926 03:42:33.068540 19896 tablet_service.cc:1514] Processing DeleteTablet for tablet 450c3ae0bf1645ef898dc0f60c113b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:33 UTC) from {username='slave'} at 127.0.0.1:48140
I20260926 03:42:33.068718 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19703
I20260926 03:42:33.069188 20133 raft_consensus.cc:2229] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:33.070020 20133 raft_consensus.cc:2258] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:33.072937 20133 ts_tablet_manager.cc:1905] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:33.086328 20135 tablet_replica.cc:330] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: stopping tablet replica
I20260926 03:42:33.086427 20134 tablet_replica.cc:330] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: stopping tablet replica
I20260926 03:42:33.087042 20135 raft_consensus.cc:2229] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:33.087121 20134 raft_consensus.cc:2229] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:33.087553 20133 ts_tablet_manager.cc:1918] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:33.087743 20133 log.cc:1196] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/450c3ae0bf1645ef898dc0f60c113b3e
I20260926 03:42:33.087821 20135 raft_consensus.cc:2258] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:33.088032 20134 raft_consensus.cc:2258] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:33.088333 20133 ts_tablet_manager.cc:1939] T 450c3ae0bf1645ef898dc0f60c113b3e P c4cd9b34e6b840e8a6b2cd89aa096e85: Deleting consensus metadata
I20260926 03:42:33.090440 20134 ts_tablet_manager.cc:1905] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:33.091334 19630 catalog_manager.cc:4841] TS c4cd9b34e6b840e8a6b2cd89aa096e85 (127.1.221.129:35545): tablet 450c3ae0bf1645ef898dc0f60c113b3e (table test-table [id=431deb284eba45fabd0fedeb02451e3c]) successfully deleted
I20260926 03:42:33.091512 20135 ts_tablet_manager.cc:1905] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:33.095065 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:33.070221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.1.221.129:35545)
0926 03:42:33.070404 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:33.070410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:33.079684 (+ 9274us) server_negotiation.cc:407] Connection header received
0926 03:42:33.080023 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:33.080030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:33.080171 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:33.080252 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:33.081034 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.082253 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.085336 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.086004 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.086110 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:33.092227 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:33.092240 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:33.092258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:33.093084 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:33.093157 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:33.093188 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:33.093471 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:33.094241 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 03:42:33.094493 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:42:33.096719 19825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:33.099501 20134 ts_tablet_manager.cc:1918] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:33.099781 20134 log.cc:1196] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/450c3ae0bf1645ef898dc0f60c113b3e
I20260926 03:42:33.100409 20134 ts_tablet_manager.cc:1939] T 450c3ae0bf1645ef898dc0f60c113b3e P 37dfca623011422199b11fb56f8631ad: Deleting consensus metadata
I20260926 03:42:33.101661 20135 ts_tablet_manager.cc:1918] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:33.101873 20135 log.cc:1196] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/450c3ae0bf1645ef898dc0f60c113b3e
I20260926 03:42:33.102550 20135 ts_tablet_manager.cc:1939] T 450c3ae0bf1645ef898dc0f60c113b3e P 85f5085e0ceb40bf827818614546e614: Deleting consensus metadata
I20260926 03:42:33.102741 19630 catalog_manager.cc:4841] TS 37dfca623011422199b11fb56f8631ad (127.1.221.131:36407): tablet 450c3ae0bf1645ef898dc0f60c113b3e (table test-table [id=431deb284eba45fabd0fedeb02451e3c]) successfully deleted
I20260926 03:42:33.104805 19628 catalog_manager.cc:4841] TS 85f5085e0ceb40bf827818614546e614 (127.1.221.130:35921): tablet 450c3ae0bf1645ef898dc0f60c113b3e (table test-table [id=431deb284eba45fabd0fedeb02451e3c]) successfully deleted
I20260926 03:42:33.246742 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19703
I20260926 03:42:33.262980 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19834
I20260926 03:42:33.270314 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:33.263866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.1.221.130:35921)
0926 03:42:33.264045 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:33.264051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:33.264070 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:42:33.264268 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:33.264280 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:33.264386 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:33.264478 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:33.264886 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.265828 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.267120 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.267633 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.267746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:33.268831 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:33.268842 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:33.268855 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:33.269470 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:33.269526 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:33.269547 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:33.269641 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:33.269807 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 03:42:33.269910 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:42:33.271193 19956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:33.383807 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19834
I20260926 03:42:33.400060 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19964
I20260926 03:42:33.406927 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:33.400899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.1.221.131:36407)
0926 03:42:33.401048 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:33.401052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:33.401070 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:42:33.401186 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:33.401190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:33.401269 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:33.401344 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:33.401678 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.402395 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.403781 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:33.404233 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:33.404346 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:33.405465 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:33.405474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:33.405507 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:33.406082 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:33.406138 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:33.406159 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:33.406250 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:33.406476 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 03:42:33.406590 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:42:33.407583 20086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:33.499413 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19964
I20260926 03:42:33.513177 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 19613
I20260926 03:42:33.514413 19674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:33.625066 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 19613
2026-09-26T03:42:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3027 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:33 dist-test-slave-lqrk krb5kdc[20143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:33 dist-test-slave-lqrk krb5kdc[20143](info): set up 2 sockets
Sep 26 03:42:33 dist-test-slave-lqrk krb5kdc[20143](info): commencing operation
krb5kdc: starting...
W20260926 03:42:34.667680 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:34 dist-test-slave-lqrk krb5kdc[20143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:42:34.947032 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44237
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:35.361641 20156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:35.362082 20156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:35.362187 20156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:35.370683 20156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:35.370864 20156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:35.370916 20156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:35.370970 20156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:35.371018 20156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:35.382604 20156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:35.384526 20156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:35.386211 20156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:35.401724 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:35.401793 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:35.401691 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:35.408275 20156 server_base.cc:1034] running on GCE node
I20260926 03:42:35.409538 20156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:35.411779 20156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:35.413080 20156 hybrid_clock.cc:648] HybridClock initialized: now 1790394155413022 us; error 52 us; skew 500 ppm
I20260926 03:42:35.413596 20156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:35 dist-test-slave-lqrk krb5kdc[20143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:35.418303 20156 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:42:35.420761 20156 webserver.cc:460] Webserver started at http://127.1.221.190:40473/ using document root <none> and password file <none>
I20260926 03:42:35.421787 20156 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:35.421955 20156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:35.422348 20156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:35.426662 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "214e2096727045f089e87930e749a887"
format_stamp: "Formatted at 2026-09-26 03:42:35 on dist-test-slave-lqrk"
server_key: "1d4435bb8f8ee65eba93d173560f7f3c"
server_key_iv: "3bede8903230bc290d70b1f2444774f9"
server_key_version: "encryptionkey@0"
I20260926 03:42:35.427801 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "214e2096727045f089e87930e749a887"
format_stamp: "Formatted at 2026-09-26 03:42:35 on dist-test-slave-lqrk"
server_key: "1d4435bb8f8ee65eba93d173560f7f3c"
server_key_iv: "3bede8903230bc290d70b1f2444774f9"
server_key_version: "encryptionkey@0"
I20260926 03:42:35.435303 20156 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 03:42:35.440718 20171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:35.442680 20156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:42:35.442950 20156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "214e2096727045f089e87930e749a887"
format_stamp: "Formatted at 2026-09-26 03:42:35 on dist-test-slave-lqrk"
server_key: "1d4435bb8f8ee65eba93d173560f7f3c"
server_key_iv: "3bede8903230bc290d70b1f2444774f9"
server_key_version: "encryptionkey@0"
I20260926 03:42:35.443274 20156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:35.472091 20156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:35.477365 20156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:35.477697 20156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:35.500213 20156 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44237
I20260926 03:42:35.500268 20222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44237 every 8 connection(s)
I20260926 03:42:35.501945 20156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:35.506487 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:35.508998 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20156
I20260926 03:42:35.509203 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:35.509789 1910 external_mini_cluster.cc:1428] Setting key 376e1f91a5a4cc7490b9fb597c255516
I20260926 03:42:35.520604 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: Bootstrap starting.
I20260926 03:42:35.525185 20223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:35.526429 20223 log.cc:824] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:42:35 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:35.530114 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: No bootstrap required, opened a new log
I20260926 03:42:35.539970 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:35.512492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.1.221.190:44237)
0926 03:42:35.513588 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:35.513603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:35.513663 (+ 60us) server_negotiation.cc:407] Connection header received
0926 03:42:35.515934 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:35.515974 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:35.516605 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:35.517111 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:35.519068 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:35.520840 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:35.523023 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:35.524020 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:35.524239 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:35.529134 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:35.529149 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:35.529172 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:35.529234 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:35.534052 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:35.535209 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:35.535221 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:35.535231 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:35.535536 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:35.536014 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:35.536019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:35.536023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:35.536896 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:35.537230 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:35.537729 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 03:42:35.538291 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":509,"threads_started":1}
I20260926 03:42:35.541194 20223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } }
I20260926 03:42:35.541616 20223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:35.541733 20223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214e2096727045f089e87930e749a887, State: Initialized, Role: FOLLOWER
I20260926 03:42:35.542397 20223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } }
I20260926 03:42:35.542649 20223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:35.542790 20223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:35.542968 20223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:35.545439 20223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } }
I20260926 03:42:35.546010 20223 leader_election.cc:302] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214e2096727045f089e87930e749a887; no voters:
I20260926 03:42:35.546689 20223 leader_election.cc:288] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:35.546940 20228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:35.547942 20228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [term 1 LEADER]: Becoming Leader. State: Replica: 214e2096727045f089e87930e749a887, State: Running, Role: LEADER
I20260926 03:42:35.548759 20228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } }
I20260926 03:42:35.549306 20223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:35.552597 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214e2096727045f089e87930e749a887. Latest consensus state: current_term: 1 leader_uuid: "214e2096727045f089e87930e749a887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } } }
I20260926 03:42:35.553131 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:35.552821 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214e2096727045f089e87930e749a887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e2096727045f089e87930e749a887" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44237 } } }
I20260926 03:42:35.553368 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:35.556559 20235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:35.562458 20235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:35.575419 20235 catalog_manager.cc:1347] Generated new cluster ID: 4b62bac01a504121b234a005ddf3cffd
I20260926 03:42:35.575572 20235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:35.612211 20235 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:35.614214 20235 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:35.624615 20235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: Generated new TSK 0
I20260926 03:42:35.625658 20235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:42:35.707312 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:36.112489 20249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:36.112797 20249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:36.112892 20249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:36.120342 20249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:36.120476 20249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:36.120597 20249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:36.130743 20249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:36.132103 20249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:36.133455 20249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:36.147295 20255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:36.147254 20254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:36.147506 20257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:36.148005 20249 server_base.cc:1034] running on GCE node
I20260926 03:42:36.148710 20249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:36.149993 20249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:36.151229 20249 hybrid_clock.cc:648] HybridClock initialized: now 1790394156151194 us; error 48 us; skew 500 ppm
I20260926 03:42:36.151624 20249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:36 dist-test-slave-lqrk krb5kdc[20143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:36.160743 20249 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:42:36.162415 20249 webserver.cc:460] Webserver started at http://127.1.221.129:34291/ using document root <none> and password file <none>
I20260926 03:42:36.163136 20249 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:36.163239 20249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:36.163596 20249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:36.166627 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "38f668144fec4150bee153a79e1c5bd6"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "def086d4fd3e63bebda940d8b31ce660"
server_key_iv: "866e43347e883d01138131a371cc461a"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.167439 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "38f668144fec4150bee153a79e1c5bd6"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "def086d4fd3e63bebda940d8b31ce660"
server_key_iv: "866e43347e883d01138131a371cc461a"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.172861 20249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:42:36.176723 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.178368 20249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:42:36.178546 20249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "38f668144fec4150bee153a79e1c5bd6"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "def086d4fd3e63bebda940d8b31ce660"
server_key_iv: "866e43347e883d01138131a371cc461a"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.178735 20249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:36.207198 20249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:36.212685 20249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:36.212989 20249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:36.214105 20249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:36.215663 20249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:36.215744 20249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.215834 20249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:36.215878 20249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.250507 20249 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41371
I20260926 03:42:36.250540 20376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41371 every 8 connection(s)
I20260926 03:42:36.251897 20249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:36.255096 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20249
I20260926 03:42:36.255288 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:36.255826 1910 external_mini_cluster.cc:1428] Setting key f4daacfed714499497836af29936cc4a
Sep 26 03:42:36 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:36.273156 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:36.254627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:51701 (local address 127.1.221.190:44237)
0926 03:42:36.255161 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:36.255167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:36.256250 (+ 1083us) server_negotiation.cc:407] Connection header received
0926 03:42:36.258439 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:36.258446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:36.258533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:36.258662 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:36.260325 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.261312 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.263012 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.263660 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.263799 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:36.267518 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:36.267525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:36.267529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:36.267573 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:36.270038 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:36.270839 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:36.270845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:36.270849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:36.270985 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:36.271425 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:36.271428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:36.271431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:36.271701 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:36.271837 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:36.272568 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 03:42:36.272810 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":258,"threads_started":1}
I20260926 03:42:36.274150 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:36.255167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44237 (local address 127.1.221.129:51701)
0926 03:42:36.256060 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:36.256108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:36.258134 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:36.258862 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:36.258871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:36.259493 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:36.260126 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:36.260146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.261495 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.261499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:36.262842 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:36.262860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.263875 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.263880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:36.264090 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:36.264813 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:36.264839 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:36.267219 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:36.270216 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:36.270227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:36.270245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:36.270713 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:36.271091 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:36.271096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:36.271100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:36.271291 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:36.271834 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:36.271844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:36.272191 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 03:42:36.272457 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 03:42:36.273034 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":362,"threads_started":1}
I20260926 03:42:36.275904 20377 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44237
I20260926 03:42:36.276288 20377 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:36.277241 20377 heartbeater.cc:511] Master 127.1.221.190:44237 requested a full tablet report, sending...
I20260926 03:42:36.279836 20188 ts_manager.cc:194] Registered new tserver with Master: 38f668144fec4150bee153a79e1c5bd6 (127.1.221.129:41371)
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260926 03:42:36.281422 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:51701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:42:36.311188 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:36.646874 20383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:36.647190 20383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:36.647306 20383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:36.655123 20383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:36.655254 20383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:36.655431 20383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:36.665762 20383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:36.667142 20383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:36.668475 20383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:36.681823 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:36.681893 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:36.682250 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:36.682839 20383 server_base.cc:1034] running on GCE node
I20260926 03:42:36.683575 20383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:36.684594 20383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:36.685859 20383 hybrid_clock.cc:648] HybridClock initialized: now 1790394156685780 us; error 67 us; skew 500 ppm
I20260926 03:42:36.686261 20383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:36 dist-test-slave-lqrk krb5kdc[20143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:36.689504 20383 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:42:36.691052 20383 webserver.cc:460] Webserver started at http://127.1.221.130:33043/ using document root <none> and password file <none>
I20260926 03:42:36.691804 20383 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:36.691916 20383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:36.692183 20383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:36.695261 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "8b1a3540681341e7b94cda967a0274ba"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "572aedbfbcfb87fb0afa76fe73a47c88"
server_key_iv: "35411c0952c6243312eb9e30d8a4c86c"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.696095 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "8b1a3540681341e7b94cda967a0274ba"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "572aedbfbcfb87fb0afa76fe73a47c88"
server_key_iv: "35411c0952c6243312eb9e30d8a4c86c"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.701431 20383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:42:36.706017 20398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.707760 20383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:36.707976 20383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "8b1a3540681341e7b94cda967a0274ba"
format_stamp: "Formatted at 2026-09-26 03:42:36 on dist-test-slave-lqrk"
server_key: "572aedbfbcfb87fb0afa76fe73a47c88"
server_key_iv: "35411c0952c6243312eb9e30d8a4c86c"
server_key_version: "encryptionkey@0"
I20260926 03:42:36.708204 20383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:36.747687 20383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:36.752408 20383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:36.752709 20383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:36.754432 20383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:36.755939 20383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:36.756039 20383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.756163 20383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:36.756218 20383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:36.789337 20383 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:33937
I20260926 03:42:36.789426 20510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:33937 every 8 connection(s)
I20260926 03:42:36.790663 20383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:36.795753 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20383
I20260926 03:42:36.795908 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:36.796304 1910 external_mini_cluster.cc:1428] Setting key 7d00c79596d1add120d05cd4598e56a2
Sep 26 03:42:36 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:36.811602 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:36.793235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39039 (local address 127.1.221.190:44237)
0926 03:42:36.793724 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:36.793729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:36.794672 (+ 943us) server_negotiation.cc:407] Connection header received
0926 03:42:36.796814 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:36.796821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:36.796913 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:36.797007 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:36.798973 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.799720 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.801293 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.801717 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.801827 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:36.805732 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:36.805737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:36.805741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:36.805778 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:36.808337 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:36.809165 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:36.809170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:36.809172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:36.809301 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:36.809763 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:36.809767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:36.809771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:36.810064 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:36.810207 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:36.810934 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 03:42:36.811187 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260926 03:42:36.812727 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:36.793736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44237 (local address 127.1.221.130:39039)
0926 03:42:36.794491 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:36.794543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:36.796525 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:36.797283 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:36.797297 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:36.798085 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:36.798793 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:36.798815 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.799870 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.799874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:36.801128 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:36.801142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:36.801961 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:36.801969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:36.802201 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:36.802942 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:36.802967 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:36.805455 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:36.808533 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:36.808543 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:36.808562 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:36.809040 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:36.809435 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:36.809440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:36.809444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:36.809651 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:36.810196 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:36.810205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:36.810574 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 03:42:36.810834 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 03:42:36.811457 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":300,"threads_started":1}
I20260926 03:42:36.814267 20511 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44237
I20260926 03:42:36.814587 20511 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:36.815372 20511 heartbeater.cc:511] Master 127.1.221.190:44237 requested a full tablet report, sending...
I20260926 03:42:36.817011 20188 ts_manager.cc:194] Registered new tserver with Master: 8b1a3540681341e7b94cda967a0274ba (127.1.221.130:33937)
I20260926 03:42:36.818202 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:39039
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:42:36.851058 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:37.189699 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:37.190101 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:37.190267 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:37.198530 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:37.198706 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:37.198875 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:37.210372 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:37.211930 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:37.213634 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:37.227984 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:37.227984 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:37.228124 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:37.229028 20517 server_base.cc:1034] running on GCE node
I20260926 03:42:37.229719 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:37.230782 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:37.232024 20517 hybrid_clock.cc:648] HybridClock initialized: now 1790394157231981 us; error 41 us; skew 500 ppm
I20260926 03:42:37.232383 20517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:37 dist-test-slave-lqrk krb5kdc[20143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:37.235955 20517 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:42:37.237634 20517 webserver.cc:460] Webserver started at http://127.1.221.131:45151/ using document root <none> and password file <none>
I20260926 03:42:37.238433 20517 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:37.238576 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:37.238906 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:37.242966 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "f8d62b83abaa4c7bb1870a33d2308236"
format_stamp: "Formatted at 2026-09-26 03:42:37 on dist-test-slave-lqrk"
server_key: "30dca7104d5452e57da3f7b2d6e7b514"
server_key_iv: "6d4f7e7b405a4961ea9c85f331868dab"
server_key_version: "encryptionkey@0"
I20260926 03:42:37.244078 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "f8d62b83abaa4c7bb1870a33d2308236"
format_stamp: "Formatted at 2026-09-26 03:42:37 on dist-test-slave-lqrk"
server_key: "30dca7104d5452e57da3f7b2d6e7b514"
server_key_iv: "6d4f7e7b405a4961ea9c85f331868dab"
server_key_version: "encryptionkey@0"
I20260926 03:42:37.250663 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:42:37.256577 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:37.258412 20517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:42:37.258591 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "f8d62b83abaa4c7bb1870a33d2308236"
format_stamp: "Formatted at 2026-09-26 03:42:37 on dist-test-slave-lqrk"
server_key: "30dca7104d5452e57da3f7b2d6e7b514"
server_key_iv: "6d4f7e7b405a4961ea9c85f331868dab"
server_key_version: "encryptionkey@0"
I20260926 03:42:37.258798 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:37.285183 20377 heartbeater.cc:503] Master 127.1.221.190:44237 was elected leader, sending a full tablet report...
I20260926 03:42:37.293741 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:37.298337 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:37.298631 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:37.300565 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:37.304491 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:37.304613 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:37.304768 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:37.304828 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:37.341939 20517 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:42391
I20260926 03:42:37.341972 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:42391 every 8 connection(s)
I20260926 03:42:37.343461 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:37.346516 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20517
I20260926 03:42:37.346710 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:37.347270 1910 external_mini_cluster.cc:1428] Setting key 1af68d3a677e78cf5789dd98fccd9f3e
Sep 26 03:42:37 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:37.366050 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.346244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54871 (local address 127.1.221.190:44237)
0926 03:42:37.346813 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.346820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.347873 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 03:42:37.350373 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.350379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.350466 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.350565 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:37.352415 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.353212 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.354990 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.355429 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.355548 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.359449 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:37.359458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:37.359462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:37.359503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:37.362466 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.363478 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.363484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.363499 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.363633 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.364146 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.364150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.364152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.364446 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:37.364605 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.365378 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.365637 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":266,"threads_started":1}
I20260926 03:42:37.367156 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.346700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44237 (local address 127.1.221.131:54871)
0926 03:42:37.347652 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.347729 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.350065 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.350886 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.350899 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.351556 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:37.352222 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.352243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.353387 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.353392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.354815 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.354840 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.355663 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.355670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.356081 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.356776 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:37.356801 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:37.359124 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:37.362617 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:37.362625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:37.362642 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:37.363345 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:37.363742 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:37.363748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:37.363752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:37.364025 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:37.364582 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:37.364594 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:37.364984 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.365241 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.365870 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":308,"threads_started":1}
I20260926 03:42:37.368862 20645 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44237
I20260926 03:42:37.369257 20645 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:37.370024 20645 heartbeater.cc:511] Master 127.1.221.190:44237 requested a full tablet report, sending...
I20260926 03:42:37.371711 20188 ts_manager.cc:194] Registered new tserver with Master: f8d62b83abaa4c7bb1870a33d2308236 (127.1.221.131:42391)
I20260926 03:42:37.372547 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:54871
I20260926 03:42:37.376333 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:37.395843 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.380986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.1.221.190:44237)
0926 03:42:37.381233 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.381240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.381501 (+ 261us) server_negotiation.cc:407] Connection header received
0926 03:42:37.381687 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.381692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.381860 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.381963 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:37.383571 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.384376 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.385825 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.386272 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.386385 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.389527 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:37.389538 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:37.389541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:37.389582 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:37.392393 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.393427 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.393432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.393440 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.393582 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.394163 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.394168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.394170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.394535 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:37.394747 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.395234 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.395471 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 03:42:37.400537 20188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:37.407696 20188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:37.434335 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.421928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:33937 (local address 127.0.0.1:58852)
0926 03:42:37.422897 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.422949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.423148 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.425689 (+ 2541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.425698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.425732 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:37.426670 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.426690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.428585 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.428591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.431347 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.431464 (+ 117us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.433112 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.433118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.433324 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.433379 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.433534 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.433741 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":463,"threads_started":1}
I20260926 03:42:37.435312 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.422530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.1.221.130:33937)
0926 03:42:37.424966 (+ 2436us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.424974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.425000 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:42:37.425108 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.425117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.425422 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.426289 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:37.426891 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.428417 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.431639 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.432911 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.433056 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.433277 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.433598 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.434960 (+ 1362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2271,"thread_start_us":2119,"threads_started":1}
I20260926 03:42:37.436000 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.422336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41371 (local address 127.0.0.1:57158)
0926 03:42:37.424687 (+ 2351us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.424708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.424950 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.425985 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.425991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.426015 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:37.427480 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.427500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.429513 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.429519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.432167 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.432182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.434352 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.434359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.434650 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.434688 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.435568 (+ 880us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.435688 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2221,"thread_start_us":4351,"threads_started":1}
I20260926 03:42:37.436030 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.423356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42391 (local address 127.0.0.1:58710)
0926 03:42:37.427036 (+ 3680us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.427062 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.427204 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.428122 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.428127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.428181 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:37.428600 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.428620 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.430503 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.430509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.432641 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.432652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.434776 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.434784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.435337 (+ 553us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.435361 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.435470 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.435608 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3546,"mutex_wait_us":4,"thread_start_us":985,"threads_started":1}
I20260926 03:42:37.436596 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.423144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.1.221.129:41371)
0926 03:42:37.424172 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.424183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.424826 (+ 643us) server_negotiation.cc:407] Connection header received
0926 03:42:37.425140 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.425148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.425422 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.425609 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:37.427754 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.429353 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.432394 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.433791 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.433934 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.434183 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.434891 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.435080 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":610,"threads_started":1}
I20260926 03:42:37.436771 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.423814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.1.221.131:42391)
0926 03:42:37.426610 (+ 2796us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.426619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.427492 (+ 873us) server_negotiation.cc:407] Connection header received
0926 03:42:37.427614 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.427623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.427904 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.428117 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:37.428808 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.430294 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.432785 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.434277 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.434432 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.435171 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.436255 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.436461 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2628,"thread_start_us":2458,"threads_started":1}
I20260926 03:42:37.439313 20446 tablet_service.cc:1467] Processing CreateTablet for tablet 9835ba247b3943418eac89e7f8198bbd (DEFAULT_TABLE table=test-table [id=534918602d164e9caf1966b99cd212b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:37.439990 20580 tablet_service.cc:1467] Processing CreateTablet for tablet 9835ba247b3943418eac89e7f8198bbd (DEFAULT_TABLE table=test-table [id=534918602d164e9caf1966b99cd212b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:37.439978 20312 tablet_service.cc:1467] Processing CreateTablet for tablet 9835ba247b3943418eac89e7f8198bbd (DEFAULT_TABLE table=test-table [id=534918602d164e9caf1966b99cd212b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:37.440755 20446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9835ba247b3943418eac89e7f8198bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:37.441351 20580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9835ba247b3943418eac89e7f8198bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:37.441373 20312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9835ba247b3943418eac89e7f8198bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:37.453958 20664 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Bootstrap starting.
I20260926 03:42:37.456418 20665 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Bootstrap starting.
I20260926 03:42:37.458349 20664 tablet_bootstrap.cc:654] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:37.460278 20665 tablet_bootstrap.cc:654] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:37.460337 20664 log.cc:824] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:37.461602 20665 log.cc:824] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:37.463193 20666 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Bootstrap starting.
I20260926 03:42:37.463950 20664 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: No bootstrap required, opened a new log
I20260926 03:42:37.464264 20664 ts_tablet_manager.cc:1397] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 03:42:37.464644 20665 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: No bootstrap required, opened a new log
I20260926 03:42:37.464936 20665 ts_tablet_manager.cc:1397] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:42:37.466626 20666 tablet_bootstrap.cc:654] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:37.468132 20666 log.cc:824] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:37.473924 20666 tablet_bootstrap.cc:492] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: No bootstrap required, opened a new log
I20260926 03:42:37.474227 20666 ts_tablet_manager.cc:1397] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 03:42:37.476204 20664 raft_consensus.cc:348] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.476204 20665 raft_consensus.cc:348] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.476713 20664 raft_consensus.cc:374] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:37.476713 20665 raft_consensus.cc:374] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:37.476824 20665 raft_consensus.cc:729] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f668144fec4150bee153a79e1c5bd6, State: Initialized, Role: FOLLOWER
I20260926 03:42:37.476824 20664 raft_consensus.cc:729] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1a3540681341e7b94cda967a0274ba, State: Initialized, Role: FOLLOWER
I20260926 03:42:37.477560 20664 consensus_queue.cc:260] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.477973 20665 consensus_queue.cc:260] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.479843 20511 heartbeater.cc:503] Master 127.1.221.190:44237 was elected leader, sending a full tablet report...
I20260926 03:42:37.481084 20664 ts_tablet_manager.cc:1428] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260926 03:42:37.483081 20665 ts_tablet_manager.cc:1428] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260926 03:42:37.484051 20666 raft_consensus.cc:348] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.484427 20666 raft_consensus.cc:374] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:37.484524 20666 raft_consensus.cc:729] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d62b83abaa4c7bb1870a33d2308236, State: Initialized, Role: FOLLOWER
I20260926 03:42:37.485193 20666 consensus_queue.cc:260] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.486624 20645 heartbeater.cc:503] Master 127.1.221.190:44237 was elected leader, sending a full tablet report...
I20260926 03:42:37.487432 20666 ts_tablet_manager.cc:1428] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260926 03:42:37.504442 20378 tablet.cc:2369] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:42:37.542868 20512 tablet.cc:2369] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:42:37.595369 20646 tablet.cc:2369] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:37.619599 20670 raft_consensus.cc:482] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:37.619912 20670 raft_consensus.cc:504] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.622215 20670 leader_election.cc:288] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8d62b83abaa4c7bb1870a33d2308236 (127.1.221.131:42391), 38f668144fec4150bee153a79e1c5bd6 (127.1.221.129:41371)
I20260926 03:42:37.632457 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.622914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55235 (local address 127.1.221.131:42391)
0926 03:42:37.623126 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.623141 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.624041 (+ 900us) server_negotiation.cc:407] Connection header received
0926 03:42:37.624310 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.624317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.624398 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.624493 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:37.625462 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.626605 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.629832 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.631125 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.631340 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.631450 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.632033 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.632187 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:42:37.632465 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.623118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57583 (local address 127.1.221.129:41371)
0926 03:42:37.623365 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.623371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.624071 (+ 700us) server_negotiation.cc:407] Connection header received
0926 03:42:37.624310 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.624317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.624398 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.624493 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:37.625462 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.626605 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.629832 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.631125 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.631325 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.631442 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.632033 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.632187 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:42:37.632663 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.623011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41371 (local address 127.1.221.130:57583)
0926 03:42:37.623953 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.623968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.624130 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.624548 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.624554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.624586 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:37.625234 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.625266 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.626792 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.626798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.629572 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.629595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.631500 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.631507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.631785 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.631842 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.631956 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.632142 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":324,"spinlock_wait_cycles":26368,"thread_start_us":455,"threads_started":1}
I20260926 03:42:37.633054 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.622634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:42391 (local address 127.1.221.130:55235)
0926 03:42:37.623870 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:37.623893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:37.624082 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:37.624548 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:37.624554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:37.624613 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:37.625275 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.625286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.626793 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.626797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.629577 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:37.629595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.631498 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.631507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:37.631792 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.631849 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 03:42:37.631956 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:42:37.632142 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":224,"spinlock_wait_cycles":32128,"thread_start_us":942,"threads_started":1}
I20260926 03:42:37.633675 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9835ba247b3943418eac89e7f8198bbd" candidate_uuid: "8b1a3540681341e7b94cda967a0274ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f668144fec4150bee153a79e1c5bd6" is_pre_election: true
I20260926 03:42:37.633960 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9835ba247b3943418eac89e7f8198bbd" candidate_uuid: "8b1a3540681341e7b94cda967a0274ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d62b83abaa4c7bb1870a33d2308236" is_pre_election: true
I20260926 03:42:37.634193 20332 raft_consensus.cc:2454] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1a3540681341e7b94cda967a0274ba in term 0.
I20260926 03:42:37.634393 20600 raft_consensus.cc:2454] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1a3540681341e7b94cda967a0274ba in term 0.
I20260926 03:42:37.634928 20400 leader_election.cc:302] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f668144fec4150bee153a79e1c5bd6, 8b1a3540681341e7b94cda967a0274ba; no voters:
I20260926 03:42:37.635319 20670 raft_consensus.cc:2781] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:37.635506 20670 raft_consensus.cc:482] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:42:37.635600 20670 raft_consensus.cc:3037] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:37.637972 20670 raft_consensus.cc:504] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.638686 20670 leader_election.cc:288] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [CANDIDATE]: Term 1 election: Requested vote from peers f8d62b83abaa4c7bb1870a33d2308236 (127.1.221.131:42391), 38f668144fec4150bee153a79e1c5bd6 (127.1.221.129:41371)
I20260926 03:42:37.639300 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9835ba247b3943418eac89e7f8198bbd" candidate_uuid: "8b1a3540681341e7b94cda967a0274ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d62b83abaa4c7bb1870a33d2308236"
I20260926 03:42:37.639355 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9835ba247b3943418eac89e7f8198bbd" candidate_uuid: "8b1a3540681341e7b94cda967a0274ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f668144fec4150bee153a79e1c5bd6"
I20260926 03:42:37.639560 20600 raft_consensus.cc:3037] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:37.639624 20332 raft_consensus.cc:3037] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:37.641852 20600 raft_consensus.cc:2454] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1a3540681341e7b94cda967a0274ba in term 1.
I20260926 03:42:37.641880 20332 raft_consensus.cc:2454] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1a3540681341e7b94cda967a0274ba in term 1.
I20260926 03:42:37.642370 20401 leader_election.cc:302] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b1a3540681341e7b94cda967a0274ba, f8d62b83abaa4c7bb1870a33d2308236; no voters:
I20260926 03:42:37.642689 20670 raft_consensus.cc:2781] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:37.643332 20670 raft_consensus.cc:686] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 1 LEADER]: Becoming Leader. State: Replica: 8b1a3540681341e7b94cda967a0274ba, State: Running, Role: LEADER
I20260926 03:42:37.643879 20670 consensus_queue.cc:237] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } }
I20260926 03:42:37.648458 20188 catalog_manager.cc:5495] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b1a3540681341e7b94cda967a0274ba (127.1.221.130). New cstate: current_term: 1 leader_uuid: "8b1a3540681341e7b94cda967a0274ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a3540681341e7b94cda967a0274ba" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 } health_report { overall_health: UNKNOWN } } }
I20260926 03:42:37.682847 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.676722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.1.221.130:33937)
0926 03:42:37.677210 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.677216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.677237 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:42:37.677476 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.677481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.677565 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.677700 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:37.678204 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.678984 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.680539 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.680945 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.681066 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.681667 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:37.682219 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:37.682384 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.682536 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.682589 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260926 03:42:37.691430 20332 raft_consensus.cc:1261] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 1 FOLLOWER]: Refusing update from remote peer 8b1a3540681341e7b94cda967a0274ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:37.691423 20600 raft_consensus.cc:1261] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 1 FOLLOWER]: Refusing update from remote peer 8b1a3540681341e7b94cda967a0274ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:37.692507 20675 consensus_queue.cc:1035] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d62b83abaa4c7bb1870a33d2308236" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:37.692848 20670 consensus_queue.cc:1035] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f668144fec4150bee153a79e1c5bd6" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:37.704370 20679 mvcc.cc:205] Tried to move back new op lower bound from 7333454469893959680 to 7333454469713399808. Current Snapshot: MvccSnapshot[applied={T|T < 7333454469893959680}]
I20260926 03:42:37.709625 20681 mvcc.cc:205] Tried to move back new op lower bound from 7333454469893959680 to 7333454469713399808. Current Snapshot: MvccSnapshot[applied={T|T < 7333454469893959680}]
I20260926 03:42:37.720876 20188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:37.721184 20188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:37.725389 20188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214e2096727045f089e87930e749a887: Sending DeleteTablet for 3 replicas of tablet 9835ba247b3943418eac89e7f8198bbd
I20260926 03:42:37.726946 20312 tablet_service.cc:1514] Processing DeleteTablet for tablet 9835ba247b3943418eac89e7f8198bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:37 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57158
I20260926 03:42:37.727065 20446 tablet_service.cc:1514] Processing DeleteTablet for tablet 9835ba247b3943418eac89e7f8198bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:37 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:58852
I20260926 03:42:37.727314 20580 tablet_service.cc:1514] Processing DeleteTablet for tablet 9835ba247b3943418eac89e7f8198bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:37 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:58710
I20260926 03:42:37.731722 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20249
I20260926 03:42:37.747260 20688 tablet_replica.cc:330] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: stopping tablet replica
I20260926 03:42:37.748059 20688 raft_consensus.cc:2229] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:37.749451 20689 tablet_replica.cc:330] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: stopping tablet replica
I20260926 03:42:37.749667 20688 raft_consensus.cc:2258] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:37.750661 20689 raft_consensus.cc:2229] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 26 03:42:37 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:42:37.753491 20689 raft_consensus.cc:2258] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:37.753815 20687 tablet_replica.cc:330] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: stopping tablet replica
I20260926 03:42:37.754000 20688 ts_tablet_manager.cc:1905] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:37.754257 20687 raft_consensus.cc:2229] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:37.754771 20687 raft_consensus.cc:2258] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:37.756474 20689 ts_tablet_manager.cc:1905] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:37.756893 20687 ts_tablet_manager.cc:1905] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:37.764060 20689 ts_tablet_manager.cc:1918] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:37.764062 20687 ts_tablet_manager.cc:1918] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:37.764257 20687 log.cc:1196] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/9835ba247b3943418eac89e7f8198bbd
I20260926 03:42:37.764436 20689 log.cc:1196] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/9835ba247b3943418eac89e7f8198bbd
I20260926 03:42:37.764858 20687 ts_tablet_manager.cc:1939] T 9835ba247b3943418eac89e7f8198bbd P 38f668144fec4150bee153a79e1c5bd6: Deleting consensus metadata
I20260926 03:42:37.765057 20689 ts_tablet_manager.cc:1939] T 9835ba247b3943418eac89e7f8198bbd P f8d62b83abaa4c7bb1870a33d2308236: Deleting consensus metadata
I20260926 03:42:37.765698 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.733884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.1.221.129:41371)
0926 03:42:37.734079 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.734086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.736255 (+ 2169us) server_negotiation.cc:407] Connection header received
0926 03:42:37.737204 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.737212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.737316 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.737422 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:37.741287 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.742643 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.747100 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.747762 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.747891 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.755195 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:37.755219 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:37.755238 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:37.755339 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:37.759636 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.762688 (+ 3052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.762697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.762700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.762904 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.763401 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.763406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.763409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.763954 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:37.764289 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.764967 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.765277 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:42:37.766952 20173 catalog_manager.cc:4841] TS 38f668144fec4150bee153a79e1c5bd6 (127.1.221.129:41371): tablet 9835ba247b3943418eac89e7f8198bbd (table test-table [id=534918602d164e9caf1966b99cd212b0]) successfully deleted
I20260926 03:42:37.767086 20174 catalog_manager.cc:4841] TS f8d62b83abaa4c7bb1870a33d2308236 (127.1.221.131:42391): tablet 9835ba247b3943418eac89e7f8198bbd (table test-table [id=534918602d164e9caf1966b99cd212b0]) successfully deleted
I20260926 03:42:37.767656 20372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:37.774145 20688 ts_tablet_manager.cc:1918] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:37.774364 20688 log.cc:1196] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/9835ba247b3943418eac89e7f8198bbd
I20260926 03:42:37.774948 20688 ts_tablet_manager.cc:1939] T 9835ba247b3943418eac89e7f8198bbd P 8b1a3540681341e7b94cda967a0274ba: Deleting consensus metadata
I20260926 03:42:37.776986 20174 catalog_manager.cc:4841] TS 8b1a3540681341e7b94cda967a0274ba (127.1.221.130:33937): tablet 9835ba247b3943418eac89e7f8198bbd (table test-table [id=534918602d164e9caf1966b99cd212b0]) successfully deleted
I20260926 03:42:37.878269 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20249
I20260926 03:42:37.894939 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20383
Sep 26 03:42:37 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:42:37.913489 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:37.895869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.1.221.130:33937)
0926 03:42:37.896093 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:37.896099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:37.896128 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:42:37.896269 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:37.896275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:37.896362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:37.896476 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:37.897808 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.899055 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.900596 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:37.901289 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:37.901430 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:37.906508 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:37.906526 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:37.906543 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:37.906587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:37.910131 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.911146 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.911154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.911156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.911322 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:37.911771 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:37.911775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:37.911778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:37.912258 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:37.912533 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:37.912937 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:42:37.913152 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:42:37.914286 20506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:38.039892 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20383
I20260926 03:42:38.059270 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20517
Sep 26 03:42:38 dist-test-slave-lqrk krb5kdc[20143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:42:38.084529 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:38.060737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.1.221.131:42391)
0926 03:42:38.060952 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:38.060964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:38.060988 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:42:38.061156 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:38.061161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:38.061235 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:38.061625 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:38.064918 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:38.066054 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:38.067951 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:38.068745 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:38.069014 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:38.076875 (+ 7861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:38.076899 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:38.076918 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:38.076974 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:38.080465 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:38.081535 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:38.081545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:38.081548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:38.081775 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:38.082286 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:38.082291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:38.082294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:38.082763 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:38.083107 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:38.083627 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 03:42:38.084029 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:42:38.085928 20640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:38.182014 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20517
I20260926 03:42:38.195990 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20156
I20260926 03:42:38.197139 20218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:38.319094 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20156
2026-09-26T03:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4692 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T03:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:38Z Disabled control of system clock
I20260926 03:42:38.394531 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34875
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:38.779708 20698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:38.780042 20698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:38.780126 20698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:38.787644 20698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:38.787766 20698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:38.787802 20698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:38.787849 20698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:38.787891 20698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:38.799748 20698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:38.801060 20698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:38.802937 20698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:38.815518 20703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:38.815555 20704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:38.815507 20706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:38.816757 20698 server_base.cc:1034] running on GCE node
I20260926 03:42:38.817631 20698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:38.819195 20698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:38.820437 20698 hybrid_clock.cc:648] HybridClock initialized: now 1790394158820372 us; error 62 us; skew 500 ppm
I20260926 03:42:38.820824 20698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:38.823323 20698 webserver.cc:460] Webserver started at http://127.1.221.190:42341/ using document root <none> and password file <none>
I20260926 03:42:38.824134 20698 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:38.824229 20698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:38.824522 20698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:38.827687 20698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "2a947839034b4cbcbf8ccb77340f12d4"
format_stamp: "Formatted at 2026-09-26 03:42:38 on dist-test-slave-lqrk"
server_key: "b7392a62325221395f3ff63b8148efdf"
server_key_iv: "049b6907f8a35431ee81cfc407a9385e"
server_key_version: "encryptionkey@0"
I20260926 03:42:38.828502 20698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "2a947839034b4cbcbf8ccb77340f12d4"
format_stamp: "Formatted at 2026-09-26 03:42:38 on dist-test-slave-lqrk"
server_key: "b7392a62325221395f3ff63b8148efdf"
server_key_iv: "049b6907f8a35431ee81cfc407a9385e"
server_key_version: "encryptionkey@0"
I20260926 03:42:38.834048 20698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:42:38.837954 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:38.839558 20698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:38.839737 20698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "2a947839034b4cbcbf8ccb77340f12d4"
format_stamp: "Formatted at 2026-09-26 03:42:38 on dist-test-slave-lqrk"
server_key: "b7392a62325221395f3ff63b8148efdf"
server_key_iv: "049b6907f8a35431ee81cfc407a9385e"
server_key_version: "encryptionkey@0"
I20260926 03:42:38.839956 20698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:38.852460 20698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:38.853718 20698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:38.854171 20698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:38.876655 20698 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34875
I20260926 03:42:38.876679 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34875 every 8 connection(s)
I20260926 03:42:38.878314 20698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:38.881515 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20698
I20260926 03:42:38.881698 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:38.882265 1910 external_mini_cluster.cc:1428] Setting key 9d13004818780b137515dc11ab62c5f5
I20260926 03:42:38.882849 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:38.897051 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: Bootstrap starting.
I20260926 03:42:38.897500 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:38.884413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.1.221.190:34875)
0926 03:42:38.886090 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:38.886102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:38.886154 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:42:38.888087 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:38.888112 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:38.888642 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:38.889060 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:38.889677 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:38.891035 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:38.892499 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:38.893260 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:38.893472 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:38.894717 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:38.894726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:38.894743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:38.895491 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:38.895577 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:38.895606 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:38.895721 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:38.895954 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 03:42:38.896295 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":481,"threads_started":1}
I20260926 03:42:38.901360 20764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:38.902426 20764 log.cc:824] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:38.905673 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: No bootstrap required, opened a new log
I20260926 03:42:38.913589 20764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } }
I20260926 03:42:38.914162 20764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:38.914263 20764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a947839034b4cbcbf8ccb77340f12d4, State: Initialized, Role: FOLLOWER
I20260926 03:42:38.914920 20764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } }
I20260926 03:42:38.915221 20764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:38.915367 20764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:38.915555 20764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:38.917743 20764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } }
I20260926 03:42:38.918253 20764 leader_election.cc:302] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a947839034b4cbcbf8ccb77340f12d4; no voters:
I20260926 03:42:38.918933 20764 leader_election.cc:288] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:38.919126 20769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:38.919839 20769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [term 1 LEADER]: Becoming Leader. State: Replica: 2a947839034b4cbcbf8ccb77340f12d4, State: Running, Role: LEADER
I20260926 03:42:38.920356 20769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } }
I20260926 03:42:38.921033 20764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:38.922955 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a947839034b4cbcbf8ccb77340f12d4. Latest consensus state: current_term: 1 leader_uuid: "2a947839034b4cbcbf8ccb77340f12d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } } }
I20260926 03:42:38.923436 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:38.923331 20770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a947839034b4cbcbf8ccb77340f12d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a947839034b4cbcbf8ccb77340f12d4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34875 } } }
I20260926 03:42:38.923653 20770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:38.925047 20777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:38.930816 20777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:38.940981 20777 catalog_manager.cc:1347] Generated new cluster ID: a36b1855d6d74a158b8009c005d21098
I20260926 03:42:38.941118 20777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:38.973739 20777 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:38.975157 20777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:38.984656 20777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: Generated new TSK 0
I20260926 03:42:38.985657 20777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:38.998968 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--builtin_ntp_servers=127.1.221.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:39.363065 20788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:39.363523 20788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:39.363639 20788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:39.371773 20788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:39.371901 20788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:39.372015 20788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:39.382644 20788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:39.384016 20788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:39.385794 20788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:39.400218 20794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:39.400290 20796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:39.400254 20793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:39.401233 20788 server_base.cc:1034] running on GCE node
I20260926 03:42:39.401984 20788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:39.403069 20788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:39.404304 20788 hybrid_clock.cc:648] HybridClock initialized: now 1790394159404224 us; error 76 us; skew 500 ppm
I20260926 03:42:39.404664 20788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:39.407312 20788 webserver.cc:460] Webserver started at http://127.1.221.129:38083/ using document root <none> and password file <none>
I20260926 03:42:39.408092 20788 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:39.408200 20788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:39.408509 20788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:39.411803 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5697e92832344cb68979ed38e37dfe25"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "c17a7c1bdf9fbe9f1af2df60c2211c77"
server_key_iv: "684e7816bec2fcab01c19e08d9f32d76"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.412688 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5697e92832344cb68979ed38e37dfe25"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "c17a7c1bdf9fbe9f1af2df60c2211c77"
server_key_iv: "684e7816bec2fcab01c19e08d9f32d76"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.418517 20788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:42:39.422662 20802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:39.425006 20788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:42:39.425222 20788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "5697e92832344cb68979ed38e37dfe25"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "c17a7c1bdf9fbe9f1af2df60c2211c77"
server_key_iv: "684e7816bec2fcab01c19e08d9f32d76"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.425465 20788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:39.457310 20788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:39.458228 20788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:39.458527 20788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:39.459640 20788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:39.461264 20788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:39.461364 20788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:39.461483 20788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:39.461555 20788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:39.496392 20788 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41847
I20260926 03:42:39.496382 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41847 every 8 connection(s)
I20260926 03:42:39.498054 20788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:39.506556 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20788
I20260926 03:42:39.506759 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:39.507504 1910 external_mini_cluster.cc:1428] Setting key eb505631f5b594b530d8f54ae80b365d
I20260926 03:42:39.511056 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--builtin_ntp_servers=127.1.221.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:39.513485 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:39.500944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59983 (local address 127.1.221.190:34875)
0926 03:42:39.501467 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:39.501472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:39.503026 (+ 1554us) server_negotiation.cc:407] Connection header received
0926 03:42:39.505055 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:39.505060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:39.505144 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:39.505256 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:39.506805 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:39.507633 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:39.509941 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:39.510404 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:39.510513 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:39.512371 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:39.512378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:39.512381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:39.512403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:39.512451 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:39.512465 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:39.512541 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:39.513083 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 03:42:39.513192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260926 03:42:39.515638 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:39.501586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34875 (local address 127.1.221.129:59983)
0926 03:42:39.502807 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:39.502855 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:39.504687 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:39.505584 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:39.505594 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:39.505702 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:39.506521 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:39.506554 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:39.507877 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:39.507883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:39.509637 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:39.509654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:39.510692 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:39.510701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:39.511067 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:39.511907 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:39.511911 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:39.511977 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:39.512112 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:39.512565 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:39.512575 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:39.512585 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:42:39.513037 (+ 452us) client_negotiation.cc:240] Negotiation successful
0926 03:42:39.513865 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":457,"threads_started":1}
I20260926 03:42:39.517510 20915 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34875
I20260926 03:42:39.517890 20915 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:39.518935 20915 heartbeater.cc:511] Master 127.1.221.190:34875 requested a full tablet report, sending...
I20260926 03:42:39.522329 20729 ts_manager.cc:194] Registered new tserver with Master: 5697e92832344cb68979ed38e37dfe25 (127.1.221.129:41847)
I20260926 03:42:39.524190 20729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:59983
W20260926 03:42:39.895515 20919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:39.895848 20919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:39.895961 20919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:39.904251 20919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:39.904374 20919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:39.904513 20919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:39.914405 20919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:39.915906 20919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:39.917683 20919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:39.932154 20924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:39.932263 20925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:39.932205 20927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:39.933736 20919 server_base.cc:1034] running on GCE node
I20260926 03:42:39.934743 20919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:39.936329 20919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:39.937629 20919 hybrid_clock.cc:648] HybridClock initialized: now 1790394159937547 us; error 56 us; skew 500 ppm
I20260926 03:42:39.938103 20919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:39.941905 20919 webserver.cc:460] Webserver started at http://127.1.221.130:35407/ using document root <none> and password file <none>
I20260926 03:42:39.942901 20919 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:39.943027 20919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:39.943449 20919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:39.947940 20919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "bbf5fbcb0a4c4c07a4fd809f95898176"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "6d860617c1ab7df38ad7d3a77e28201d"
server_key_iv: "afbb4ca9faa4898222349fd92fb6ce02"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.949012 20919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "bbf5fbcb0a4c4c07a4fd809f95898176"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "6d860617c1ab7df38ad7d3a77e28201d"
server_key_iv: "afbb4ca9faa4898222349fd92fb6ce02"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.956157 20919 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260926 03:42:39.961339 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:39.963346 20919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260926 03:42:39.963762 20919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "bbf5fbcb0a4c4c07a4fd809f95898176"
format_stamp: "Formatted at 2026-09-26 03:42:39 on dist-test-slave-lqrk"
server_key: "6d860617c1ab7df38ad7d3a77e28201d"
server_key_iv: "afbb4ca9faa4898222349fd92fb6ce02"
server_key_version: "encryptionkey@0"
I20260926 03:42:39.964053 20919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:39.986637 20919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:39.987841 20919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:39.988201 20919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:39.989511 20919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:39.991465 20919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:39.991577 20919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:39.991703 20919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:39.991775 20919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:40.030778 20919 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:39189
I20260926 03:42:40.030802 21045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:39189 every 8 connection(s)
I20260926 03:42:40.032527 20919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:40.038029 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 20919
I20260926 03:42:40.038223 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:40.038688 1910 external_mini_cluster.cc:1428] Setting key 47ac2c3deb8157d9a0fdf98d54020a37
I20260926 03:42:40.042340 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--builtin_ntp_servers=127.1.221.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:40.052573 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.035911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:40479 (local address 127.1.221.190:34875)
0926 03:42:40.036640 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.036648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.038540 (+ 1892us) server_negotiation.cc:407] Connection header received
0926 03:42:40.041564 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.041571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.041704 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.041851 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:40.044730 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.045712 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.048399 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.048935 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.049103 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.051307 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:40.051313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:40.051317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:40.051347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:40.051460 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:40.051480 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:40.051576 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.051923 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.052109 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":307,"threads_started":1}
I20260926 03:42:40.054567 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.036821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34875 (local address 127.1.221.130:40479)
0926 03:42:40.038267 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.038342 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.041065 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.042895 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.042913 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.043288 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:40.044413 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.044451 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.045975 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.045980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.048116 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.048145 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.049279 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.049289 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.049623 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.050702 (+ 1079us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:40.050706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:40.050780 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:40.050928 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:40.051603 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:40.051618 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:40.051627 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.051863 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.052591 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":427,"threads_started":1}
I20260926 03:42:40.057375 21046 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34875
I20260926 03:42:40.058089 21046 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:40.059563 21046 heartbeater.cc:511] Master 127.1.221.190:34875 requested a full tablet report, sending...
I20260926 03:42:40.062822 20729 ts_manager.cc:194] Registered new tserver with Master: bbf5fbcb0a4c4c07a4fd809f95898176 (127.1.221.130:39189)
I20260926 03:42:40.064175 20729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:40479
W20260926 03:42:40.429972 21050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:40.430402 21050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:40.430544 21050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:40.440872 21050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:40.441015 21050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:40.441136 21050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:40.451328 21050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:40.452797 21050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:40.454735 21050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:40.473470 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:40.473515 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:40.473737 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:40.475307 21050 server_base.cc:1034] running on GCE node
I20260926 03:42:40.476382 21050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:40.477900 21050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:40.479213 21050 hybrid_clock.cc:648] HybridClock initialized: now 1790394160479144 us; error 58 us; skew 500 ppm
I20260926 03:42:40.479808 21050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:40.483196 21050 webserver.cc:460] Webserver started at http://127.1.221.131:34315/ using document root <none> and password file <none>
I20260926 03:42:40.484252 21050 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:40.484377 21050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:40.484793 21050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:40.488921 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "ab1749e119084495a72e3b8404bb68e3"
format_stamp: "Formatted at 2026-09-26 03:42:40 on dist-test-slave-lqrk"
server_key: "0fdac398da9e8cc53dcd0045e4faa438"
server_key_iv: "8fdf8206e68bacabfd82f3346a5fb62f"
server_key_version: "encryptionkey@0"
I20260926 03:42:40.490072 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "ab1749e119084495a72e3b8404bb68e3"
format_stamp: "Formatted at 2026-09-26 03:42:40 on dist-test-slave-lqrk"
server_key: "0fdac398da9e8cc53dcd0045e4faa438"
server_key_iv: "8fdf8206e68bacabfd82f3346a5fb62f"
server_key_version: "encryptionkey@0"
I20260926 03:42:40.497820 21050 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 03:42:40.503369 21064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:40.505867 21050 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:42:40.506181 21050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "ab1749e119084495a72e3b8404bb68e3"
format_stamp: "Formatted at 2026-09-26 03:42:40 on dist-test-slave-lqrk"
server_key: "0fdac398da9e8cc53dcd0045e4faa438"
server_key_iv: "8fdf8206e68bacabfd82f3346a5fb62f"
server_key_version: "encryptionkey@0"
I20260926 03:42:40.506605 21050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:40.531909 20915 heartbeater.cc:503] Master 127.1.221.190:34875 was elected leader, sending a full tablet report...
I20260926 03:42:40.541707 21050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:40.543040 21050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:40.543422 21050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:40.544905 21050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:40.546960 21050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:40.547070 21050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:40.547194 21050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:40.547261 21050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:40.591471 21050 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36129
I20260926 03:42:40.591843 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36129 every 8 connection(s)
I20260926 03:42:40.593217 21050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:40.596410 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21050
I20260926 03:42:40.596588 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:40.597123 1910 external_mini_cluster.cc:1428] Setting key 25f0e9b2f0b4a6ef17e72a6fced08e12
I20260926 03:42:40.610386 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.596575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:40553 (local address 127.1.221.190:34875)
0926 03:42:40.597268 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.597280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.599232 (+ 1952us) server_negotiation.cc:407] Connection header received
0926 03:42:40.601615 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.601621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.601722 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.601882 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:40.603309 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.604330 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.606419 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.606978 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.607163 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.609116 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:40.609124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:40.609128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:40.609160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:40.609226 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:40.609244 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:40.609361 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.609784 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.609949 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":332,"threads_started":1}
I20260926 03:42:40.612432 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.597495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34875 (local address 127.1.221.131:40553)
0926 03:42:40.599017 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.599089 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.601217 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.602203 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.602213 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.602329 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:40.603022 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.603041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.604595 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.604601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.606131 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.606151 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.607328 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.607335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.607651 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.608517 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:40.608520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:40.608583 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:40.608692 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:40.609415 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:40.609427 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:40.609436 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.609678 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.610377 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":493,"threads_started":1}
I20260926 03:42:40.614464 21177 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34875
I20260926 03:42:40.614984 21177 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:40.616169 21177 heartbeater.cc:511] Master 127.1.221.190:34875 requested a full tablet report, sending...
I20260926 03:42:40.618428 20729 ts_manager.cc:194] Registered new tserver with Master: ab1749e119084495a72e3b8404bb68e3 (127.1.221.131:36129)
I20260926 03:42:40.619453 20729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:40553
I20260926 03:42:40.629318 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:40.641615 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.634282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.1.221.190:34875)
0926 03:42:40.634612 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.634620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.634931 (+ 311us) server_negotiation.cc:407] Connection header received
0926 03:42:40.635171 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.635176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.635269 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.635368 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:40.636076 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.636941 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.638552 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.639008 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.639149 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.640688 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:40.640696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:40.640702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:40.640732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:40.640789 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:40.640811 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:40.640962 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.641210 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.641288 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 03:42:40.647763 20729 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:40.656580 20729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:40.689010 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.674883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36129 (local address 127.0.0.1:60738)
0926 03:42:40.676417 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.676475 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.676738 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.678448 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.678462 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.678515 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:40.679144 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.679161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.682187 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.682192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.684922 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.684941 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.687769 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.687777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.688052 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.688121 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.688302 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.688577 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":3265,"threads_started":1}
I20260926 03:42:40.690466 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.675366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.1.221.131:36129)
0926 03:42:40.677620 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.677632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.677677 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:42:40.677801 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.677812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.678132 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.678361 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:40.679466 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.681948 (+ 2482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.685241 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.687007 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.687201 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.687601 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.689770 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.690045 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1988,"thread_start_us":1774,"threads_started":1}
I20260926 03:42:40.693674 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.676183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.1.221.129:41847)
0926 03:42:40.677948 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.677970 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.678673 (+ 703us) server_negotiation.cc:407] Connection header received
0926 03:42:40.679047 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.679063 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.679449 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.679687 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:40.682229 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.684117 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.688929 (+ 4812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.690694 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.690826 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.691199 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.692984 (+ 1785us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.693237 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1535,"thread_start_us":1635,"threads_started":1}
I20260926 03:42:40.694553 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.675361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41847 (local address 127.0.0.1:51172)
0926 03:42:40.678477 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.678526 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.678801 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.680028 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.680032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.680065 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:40.681998 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.682017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.684376 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.684381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.688629 (+ 4248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.688648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.691478 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.691486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.691717 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.691753 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.694044 (+ 2291us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.694211 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2959,"thread_start_us":1012,"threads_started":1}
I20260926 03:42:40.699357 21112 tablet_service.cc:1467] Processing CreateTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc (DEFAULT_TABLE table=test-table [id=305b7c3e89114e2793833988d1727f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:40.699357 20850 tablet_service.cc:1467] Processing CreateTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc (DEFAULT_TABLE table=test-table [id=305b7c3e89114e2793833988d1727f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:40.700219 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.678952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39189 (local address 127.0.0.1:51814)
0926 03:42:40.689393 (+ 10441us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.689417 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.689560 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.693475 (+ 3915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.693483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.693545 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:40.694128 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.694150 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.696224 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.696229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.697676 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.697691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.699328 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.699334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.699558 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.699596 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.699694 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.699864 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10274,"thread_start_us":14839,"threads_started":1}
I20260926 03:42:40.700554 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.678956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.1.221.130:39189)
0926 03:42:40.692404 (+ 13448us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.692415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.692455 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:42:40.692615 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.692626 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.693093 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.693389 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:40.694504 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.696043 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.697907 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.699144 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.699279 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.699605 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.699752 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.699942 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13218,"thread_start_us":13078,"threads_started":1}
I20260926 03:42:40.701416 20850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5983dd9e2e6241bb89880e1986d5adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:40.701440 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5983dd9e2e6241bb89880e1986d5adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:40.704646 20981 tablet_service.cc:1467] Processing CreateTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc (DEFAULT_TABLE table=test-table [id=305b7c3e89114e2793833988d1727f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:40.706176 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5983dd9e2e6241bb89880e1986d5adcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:40.719877 21196 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Bootstrap starting.
I20260926 03:42:40.721829 21198 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Bootstrap starting.
I20260926 03:42:40.722322 21197 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Bootstrap starting.
I20260926 03:42:40.726727 21198 tablet_bootstrap.cc:654] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:40.727164 21196 tablet_bootstrap.cc:654] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:40.727161 21197 tablet_bootstrap.cc:654] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:40.728348 21198 log.cc:824] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:40.728740 21197 log.cc:824] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:40.728749 21196 log.cc:824] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:40.733088 21198 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: No bootstrap required, opened a new log
I20260926 03:42:40.733130 21196 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: No bootstrap required, opened a new log
I20260926 03:42:40.733129 21197 tablet_bootstrap.cc:492] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: No bootstrap required, opened a new log
I20260926 03:42:40.733501 21198 ts_tablet_manager.cc:1397] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 03:42:40.733484 21196 ts_tablet_manager.cc:1397] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20260926 03:42:40.733834 21197 ts_tablet_manager.cc:1397] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 03:42:40.743170 21197 raft_consensus.cc:348] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.743954 21197 raft_consensus.cc:374] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:40.744071 21197 raft_consensus.cc:729] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5697e92832344cb68979ed38e37dfe25, State: Initialized, Role: FOLLOWER
I20260926 03:42:40.745025 21197 consensus_queue.cc:260] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.746595 21198 raft_consensus.cc:348] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.746595 21196 raft_consensus.cc:348] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.747191 21198 raft_consensus.cc:374] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:40.747191 21196 raft_consensus.cc:374] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:40.747310 21196 raft_consensus.cc:729] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf5fbcb0a4c4c07a4fd809f95898176, State: Initialized, Role: FOLLOWER
I20260926 03:42:40.747305 21198 raft_consensus.cc:729] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1749e119084495a72e3b8404bb68e3, State: Initialized, Role: FOLLOWER
I20260926 03:42:40.747834 21197 ts_tablet_manager.cc:1428] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:42:40.748569 21196 consensus_queue.cc:260] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.748548 21198 consensus_queue.cc:260] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
W20260926 03:42:40.750854 20916 tablet.cc:2369] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:40.753170 21046 heartbeater.cc:503] Master 127.1.221.190:34875 was elected leader, sending a full tablet report...
I20260926 03:42:40.754261 21196 ts_tablet_manager.cc:1428] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260926 03:42:40.755548 21177 heartbeater.cc:503] Master 127.1.221.190:34875 was elected leader, sending a full tablet report...
I20260926 03:42:40.756771 21198 ts_tablet_manager.cc:1428] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s
W20260926 03:42:40.785586 21047 tablet.cc:2369] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:42:40.845372 21178 tablet.cc:2369] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:40.877785 21202 raft_consensus.cc:482] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:40.878290 21202 raft_consensus.cc:504] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.881418 21202 leader_election.cc:288] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1749e119084495a72e3b8404bb68e3 (127.1.221.131:36129), bbf5fbcb0a4c4c07a4fd809f95898176 (127.1.221.130:39189)
I20260926 03:42:40.890390 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.881909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34699 (local address 127.1.221.131:36129)
0926 03:42:40.882152 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.882158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.883114 (+ 956us) server_negotiation.cc:407] Connection header received
0926 03:42:40.883564 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.883570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.883660 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.883733 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:40.884567 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.885667 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.888365 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.889226 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.889416 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.889498 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.889962 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.890108 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:42:40.890606 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.881939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36129 (local address 127.1.221.129:34699)
0926 03:42:40.882976 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.882999 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.883272 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.883956 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.883960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.883986 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:40.884385 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.884395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.885895 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.885902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.888092 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.888109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.889504 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.889512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.889757 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.889787 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.889894 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.890064 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":51,"thread_start_us":552,"threads_started":1}
I20260926 03:42:40.892426 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5983dd9e2e6241bb89880e1986d5adcc" candidate_uuid: "5697e92832344cb68979ed38e37dfe25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1749e119084495a72e3b8404bb68e3" is_pre_election: true
I20260926 03:42:40.893601 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.883574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58839 (local address 127.1.221.130:39189)
0926 03:42:40.883795 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.883801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.885398 (+ 1597us) server_negotiation.cc:407] Connection header received
0926 03:42:40.885559 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.885574 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.885715 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.885876 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:40.886894 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.888187 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.891600 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.892508 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.892696 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.892775 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.893168 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.893316 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:42:40.893471 21132 raft_consensus.cc:2454] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5697e92832344cb68979ed38e37dfe25 in term 0.
I20260926 03:42:40.893682 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.883192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39189 (local address 127.1.221.129:58839)
0926 03:42:40.885198 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:40.885231 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:40.885384 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:40.886007 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:40.886012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:40.886044 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:40.886621 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.886640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.888484 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.888492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.891182 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:40.891227 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.892764 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.892772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:40.892988 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.893026 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:42:40.893128 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:42:40.893298 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1847,"mutex_wait_us":66,"thread_start_us":1568,"threads_started":1}
I20260926 03:42:40.894989 20805 leader_election.cc:302] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5697e92832344cb68979ed38e37dfe25, ab1749e119084495a72e3b8404bb68e3; no voters:
I20260926 03:42:40.895072 21001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5983dd9e2e6241bb89880e1986d5adcc" candidate_uuid: "5697e92832344cb68979ed38e37dfe25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" is_pre_election: true
I20260926 03:42:40.895768 21202 raft_consensus.cc:2781] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:40.895809 21001 raft_consensus.cc:2454] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5697e92832344cb68979ed38e37dfe25 in term 0.
I20260926 03:42:40.896024 21202 raft_consensus.cc:482] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:42:40.896143 21202 raft_consensus.cc:3037] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:40.899564 21202 raft_consensus.cc:504] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.900418 21202 leader_election.cc:288] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [CANDIDATE]: Term 1 election: Requested vote from peers ab1749e119084495a72e3b8404bb68e3 (127.1.221.131:36129), bbf5fbcb0a4c4c07a4fd809f95898176 (127.1.221.130:39189)
I20260926 03:42:40.901528 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5983dd9e2e6241bb89880e1986d5adcc" candidate_uuid: "5697e92832344cb68979ed38e37dfe25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1749e119084495a72e3b8404bb68e3"
I20260926 03:42:40.901554 21001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5983dd9e2e6241bb89880e1986d5adcc" candidate_uuid: "5697e92832344cb68979ed38e37dfe25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176"
I20260926 03:42:40.901908 21001 raft_consensus.cc:3037] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:40.901957 21132 raft_consensus.cc:3037] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:40.904796 21132 raft_consensus.cc:2454] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5697e92832344cb68979ed38e37dfe25 in term 1.
I20260926 03:42:40.905056 21001 raft_consensus.cc:2454] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5697e92832344cb68979ed38e37dfe25 in term 1.
I20260926 03:42:40.905661 20805 leader_election.cc:302] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5697e92832344cb68979ed38e37dfe25, ab1749e119084495a72e3b8404bb68e3; no voters:
I20260926 03:42:40.906317 21202 raft_consensus.cc:2781] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:40.907045 21202 raft_consensus.cc:686] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 1 LEADER]: Becoming Leader. State: Replica: 5697e92832344cb68979ed38e37dfe25, State: Running, Role: LEADER
I20260926 03:42:40.907889 21202 consensus_queue.cc:237] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } }
I20260926 03:42:40.914839 20728 catalog_manager.cc:5495] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5697e92832344cb68979ed38e37dfe25 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "5697e92832344cb68979ed38e37dfe25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5697e92832344cb68979ed38e37dfe25" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 } health_report { overall_health: UNKNOWN } } }
I20260926 03:42:40.947851 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:40.939997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.1.221.129:41847)
0926 03:42:40.940329 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:40.940335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:40.940362 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:42:40.940603 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:40.940664 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:40.940808 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:40.941014 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:40.941561 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.942446 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.944283 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:40.945001 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:40.945199 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:40.946023 (+ 824us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:40.946867 (+ 844us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:40.947049 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:40.947422 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 03:42:40.947496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 03:42:40.960541 21132 raft_consensus.cc:1261] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 1 FOLLOWER]: Refusing update from remote peer 5697e92832344cb68979ed38e37dfe25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:40.960721 21001 raft_consensus.cc:1261] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 1 FOLLOWER]: Refusing update from remote peer 5697e92832344cb68979ed38e37dfe25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:40.962335 21202 consensus_queue.cc:1035] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf5fbcb0a4c4c07a4fd809f95898176" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:40.962918 21207 consensus_queue.cc:1035] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1749e119084495a72e3b8404bb68e3" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:41.001372 20729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:41.001740 20729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:41.007660 20729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a947839034b4cbcbf8ccb77340f12d4: Sending DeleteTablet for 3 replicas of tablet 5983dd9e2e6241bb89880e1986d5adcc
I20260926 03:42:41.009600 21112 tablet_service.cc:1514] Processing DeleteTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:40 UTC) from {username='slave'} at 127.0.0.1:60738
I20260926 03:42:41.009600 20981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:40 UTC) from {username='slave'} at 127.0.0.1:51814
I20260926 03:42:41.009601 20850 tablet_service.cc:1514] Processing DeleteTablet for tablet 5983dd9e2e6241bb89880e1986d5adcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:40 UTC) from {username='slave'} at 127.0.0.1:51172
I20260926 03:42:41.012452 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20788
I20260926 03:42:41.020061 21222 tablet_replica.cc:330] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: stopping tablet replica
I20260926 03:42:41.021178 21222 raft_consensus.cc:2229] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:41.022001 21222 raft_consensus.cc:2258] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:41.022322 21221 tablet_replica.cc:330] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: stopping tablet replica
I20260926 03:42:41.022967 21221 raft_consensus.cc:2229] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:41.023805 21221 raft_consensus.cc:2258] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:41.025695 21222 ts_tablet_manager.cc:1905] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:41.034469 21222 ts_tablet_manager.cc:1918] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:41.035480 21222 log.cc:1196] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/5983dd9e2e6241bb89880e1986d5adcc
I20260926 03:42:41.035743 21220 tablet_replica.cc:330] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: stopping tablet replica
I20260926 03:42:41.036392 21220 raft_consensus.cc:2229] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:41.037293 21221 ts_tablet_manager.cc:1905] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:41.037308 21220 raft_consensus.cc:2258] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:41.039901 21220 ts_tablet_manager.cc:1905] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:41.045430 21221 ts_tablet_manager.cc:1918] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:41.046233 21222 ts_tablet_manager.cc:1939] T 5983dd9e2e6241bb89880e1986d5adcc P ab1749e119084495a72e3b8404bb68e3: Deleting consensus metadata
I20260926 03:42:41.048564 21221 log.cc:1196] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/5983dd9e2e6241bb89880e1986d5adcc
I20260926 03:42:41.048496 20715 catalog_manager.cc:4841] TS ab1749e119084495a72e3b8404bb68e3 (127.1.221.131:36129): tablet 5983dd9e2e6241bb89880e1986d5adcc (table test-table [id=305b7c3e89114e2793833988d1727f39]) successfully deleted
I20260926 03:42:41.049264 21221 ts_tablet_manager.cc:1939] T 5983dd9e2e6241bb89880e1986d5adcc P bbf5fbcb0a4c4c07a4fd809f95898176: Deleting consensus metadata
I20260926 03:42:41.049299 21220 ts_tablet_manager.cc:1918] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:41.049822 21220 log.cc:1196] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/5983dd9e2e6241bb89880e1986d5adcc
I20260926 03:42:41.050215 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:41.013753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.1.221.129:41847)
0926 03:42:41.014025 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:41.014032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:41.020147 (+ 6115us) server_negotiation.cc:407] Connection header received
0926 03:42:41.021885 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:41.021892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:41.022053 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:41.022148 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:41.023142 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.024765 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.027207 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.028037 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.028187 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:41.044804 (+ 16617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:41.044823 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:41.044881 (+ 58us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:41.046153 (+ 1272us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:41.046265 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:41.046313 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:41.046504 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:41.047529 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0926 03:42:41.049559 (+ 2030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:42:41.050633 21220 ts_tablet_manager.cc:1939] T 5983dd9e2e6241bb89880e1986d5adcc P 5697e92832344cb68979ed38e37dfe25: Deleting consensus metadata
I20260926 03:42:41.051509 20715 catalog_manager.cc:4841] TS bbf5fbcb0a4c4c07a4fd809f95898176 (127.1.221.130:39189): tablet 5983dd9e2e6241bb89880e1986d5adcc (table test-table [id=305b7c3e89114e2793833988d1727f39]) successfully deleted
I20260926 03:42:41.052488 20910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:41.053777 20713 catalog_manager.cc:4841] TS 5697e92832344cb68979ed38e37dfe25 (127.1.221.129:41847): tablet 5983dd9e2e6241bb89880e1986d5adcc (table test-table [id=305b7c3e89114e2793833988d1727f39]) successfully deleted
I20260926 03:42:41.189872 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20788
I20260926 03:42:41.213590 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20919
I20260926 03:42:41.230082 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:41.214771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.1.221.130:39189)
0926 03:42:41.215068 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:41.215073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:41.215101 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:42:41.215272 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:41.215277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:41.215858 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:41.216083 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:41.216839 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.217768 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.219559 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.220024 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.220584 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:41.228233 (+ 7649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:41.228245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:41.228265 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:41.228969 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:41.229033 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:41.229061 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:41.229184 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:41.229373 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 03:42:41.229512 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 03:42:41.231276 21041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:41.342586 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20919
I20260926 03:42:41.359366 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21050
I20260926 03:42:41.368163 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:41.360548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.1.221.131:36129)
0926 03:42:41.360839 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:41.360845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:41.360869 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:42:41.360947 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:41.360952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:41.361071 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:41.361219 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:41.361896 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.362700 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.364558 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:41.365039 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:41.365191 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:41.366558 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:41.366567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:41.366582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:41.367243 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:41.367341 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:41.367365 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:41.367532 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:41.367728 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 03:42:41.367838 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 03:42:41.369329 21172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:41.475334 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21050
I20260926 03:42:41.493443 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 20698
I20260926 03:42:41.494876 20759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:41.611346 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 20698
2026-09-26T03:42:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:41 dist-test-slave-lqrk krb5kdc[21230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:41 dist-test-slave-lqrk krb5kdc[21230](info): set up 2 sockets
Sep 26 03:42:41 dist-test-slave-lqrk krb5kdc[21230](info): commencing operation
krb5kdc: starting...
W20260926 03:42:42.712216 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:42 dist-test-slave-lqrk krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:42:43.028219 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34297
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:43.382967 21243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:43.383320 21243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:43.383517 21243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:43.391563 21243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:43.391765 21243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:43.391865 21243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:43.391954 21243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:43.392028 21243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:43.403173 21243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:43.404654 21243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:43.406021 21243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:43.419114 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:43.419440 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:43.419742 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:43.421144 21243 server_base.cc:1034] running on GCE node
I20260926 03:42:43.422291 21243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:43.424502 21243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:43.425861 21243 hybrid_clock.cc:648] HybridClock initialized: now 1790394163425765 us; error 80 us; skew 500 ppm
I20260926 03:42:43.426510 21243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:43 dist-test-slave-lqrk krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:43.431741 21243 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:42:43.434314 21243 webserver.cc:460] Webserver started at http://127.1.221.190:38479/ using document root <none> and password file <none>
I20260926 03:42:43.435570 21243 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:43.435745 21243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:43.436178 21243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:43.442436 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "a5f9e133459f47bb84558090284ef7ff"
format_stamp: "Formatted at 2026-09-26 03:42:43 on dist-test-slave-lqrk"
server_key: "6c4889cf1291c51246bc03ccf8cd8edb"
server_key_iv: "9b834e55ed94c43714514e8bcfc43cee"
server_key_version: "encryptionkey@0"
I20260926 03:42:43.443867 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a5f9e133459f47bb84558090284ef7ff"
format_stamp: "Formatted at 2026-09-26 03:42:43 on dist-test-slave-lqrk"
server_key: "6c4889cf1291c51246bc03ccf8cd8edb"
server_key_iv: "9b834e55ed94c43714514e8bcfc43cee"
server_key_version: "encryptionkey@0"
I20260926 03:42:43.453495 21243 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260926 03:42:43.459487 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:43.461815 21243 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:42:43.462092 21243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "a5f9e133459f47bb84558090284ef7ff"
format_stamp: "Formatted at 2026-09-26 03:42:43 on dist-test-slave-lqrk"
server_key: "6c4889cf1291c51246bc03ccf8cd8edb"
server_key_iv: "9b834e55ed94c43714514e8bcfc43cee"
server_key_version: "encryptionkey@0"
I20260926 03:42:43.462383 21243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:43.480839 21243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:43.486200 21243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:43.486557 21243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:43.507804 21243 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34297
I20260926 03:42:43.507831 21309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34297 every 8 connection(s)
I20260926 03:42:43.509613 21243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:43.511144 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21243
I20260926 03:42:43.511369 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:43.511968 1910 external_mini_cluster.cc:1428] Setting key 4662a3e538bbef386c9629e6d2e7a4f1
I20260926 03:42:43.515165 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:43.528090 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: Bootstrap starting.
Sep 26 03:42:43 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:43.532802 21310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:43.534379 21310 log.cc:824] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:43.538791 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: No bootstrap required, opened a new log
I20260926 03:42:43.542387 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:43.514175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.1.221.190:34297)
0926 03:42:43.515187 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:43.515205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:43.515269 (+ 64us) server_negotiation.cc:407] Connection header received
0926 03:42:43.517336 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:43.517369 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:43.517933 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:43.518452 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:43.520271 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:43.521853 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:43.523405 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:43.524579 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:43.524867 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:43.531092 (+ 6225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:43.531104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:43.531131 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:43.531201 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:43.535742 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:43.536967 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:43.536978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:43.536988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:43.537296 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:43.537914 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:43.537924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:43.537926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:43.538738 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:43.539135 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:43.539822 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 03:42:43.540628 (+ 806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":434,"threads_started":1}
I20260926 03:42:43.548696 21310 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } }
I20260926 03:42:43.549116 21310 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:43.549214 21310 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f9e133459f47bb84558090284ef7ff, State: Initialized, Role: FOLLOWER
I20260926 03:42:43.549856 21310 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } }
I20260926 03:42:43.550084 21310 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:43.550225 21310 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:43.550396 21310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:43.553007 21310 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } }
I20260926 03:42:43.553781 21310 leader_election.cc:302] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f9e133459f47bb84558090284ef7ff; no voters:
I20260926 03:42:43.554615 21310 leader_election.cc:288] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:43.554901 21315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:43.555938 21315 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [term 1 LEADER]: Becoming Leader. State: Replica: a5f9e133459f47bb84558090284ef7ff, State: Running, Role: LEADER
I20260926 03:42:43.556715 21315 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } }
I20260926 03:42:43.557144 21310 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:43.560499 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f9e133459f47bb84558090284ef7ff. Latest consensus state: current_term: 1 leader_uuid: "a5f9e133459f47bb84558090284ef7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } } }
I20260926 03:42:43.560997 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:43.561697 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f9e133459f47bb84558090284ef7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9e133459f47bb84558090284ef7ff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34297 } } }
I20260926 03:42:43.562168 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:43.564375 21324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:43.571700 21324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:43.584295 21324 catalog_manager.cc:1347] Generated new cluster ID: f7abba559e7b471aa857bc70d416923f
I20260926 03:42:43.584484 21324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:43.629865 21324 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:43.632100 21324 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:43.649142 21324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: Generated new TSK 0
I20260926 03:42:43.650736 21324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260926 03:42:43.741336 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:44.138062 21336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:44.138389 21336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:44.138485 21336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:44.146291 21336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:44.146422 21336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:44.146539 21336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:44.156853 21336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:44.158340 21336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:44.159776 21336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:44.174283 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:44.174513 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:44.174573 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:44.176561 21336 server_base.cc:1034] running on GCE node
I20260926 03:42:44.177227 21336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:44.178319 21336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:44.179527 21336 hybrid_clock.cc:648] HybridClock initialized: now 1790394164179469 us; error 44 us; skew 500 ppm
I20260926 03:42:44.179908 21336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:44 dist-test-slave-lqrk krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:44.183794 21336 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260926 03:42:44.186012 21336 webserver.cc:460] Webserver started at http://127.1.221.129:33591/ using document root <none> and password file <none>
I20260926 03:42:44.187317 21336 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:44.187557 21336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:44.187970 21336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:44.191505 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "937e47a2920847d6bf471eb495f03996"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "cc3d7a73b004b8cbcd85db4b4b07956a"
server_key_iv: "e24e7cf43a5247f11203a46e5535bdea"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.192478 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "937e47a2920847d6bf471eb495f03996"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "cc3d7a73b004b8cbcd85db4b4b07956a"
server_key_iv: "e24e7cf43a5247f11203a46e5535bdea"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.198653 21336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:42:44.202932 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.204604 21336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:44.204797 21336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "937e47a2920847d6bf471eb495f03996"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "cc3d7a73b004b8cbcd85db4b4b07956a"
server_key_iv: "e24e7cf43a5247f11203a46e5535bdea"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.205013 21336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:44.222620 21336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:44.227460 21336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:44.227763 21336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:44.228937 21336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:44.230595 21336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:44.230686 21336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.230834 21336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:44.230904 21336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.266535 21336 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:46549
I20260926 03:42:44.266571 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:46549 every 8 connection(s)
I20260926 03:42:44.268151 21336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:44.275441 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21336
I20260926 03:42:44.275635 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:44.276087 1910 external_mini_cluster.cc:1428] Setting key e61750599a2e92e1e7aff161612dbf40
Sep 26 03:42:44 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:44.293644 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:44.270805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39647 (local address 127.1.221.190:34297)
0926 03:42:44.271347 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:44.271353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:44.272628 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 03:42:44.274717 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:44.274724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:44.274816 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:44.274924 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:44.276972 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.277819 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.279587 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.280023 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.280149 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:44.286002 (+ 5853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:44.286009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:44.286013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:44.286055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:44.289532 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:44.290551 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:44.290556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:44.290559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:44.290712 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:44.291617 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:44.291622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:44.291625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:44.291981 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:44.292136 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:44.292998 (+ 862us) server_negotiation.cc:299] Negotiation successful
0926 03:42:44.293263 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260926 03:42:44.294548 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:44.271516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34297 (local address 127.1.221.129:39647)
0926 03:42:44.272433 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:44.272484 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:44.274405 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:44.275207 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:44.275217 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:44.275914 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:44.276742 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:44.276768 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.278012 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.278016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:44.279427 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:44.279446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.280280 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.280288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:44.280552 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:44.281521 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:44.281568 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:44.285402 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:44.289791 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:44.289801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:44.289818 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:44.290400 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:44.290944 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:44.290950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:44.290954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:44.291484 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:44.292137 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:44.292145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:44.292550 (+ 405us) client_negotiation.cc:769] Sending connection context
0926 03:42:44.292815 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 03:42:44.293279 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":343,"threads_started":1}
I20260926 03:42:44.296517 21464 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34297
I20260926 03:42:44.296878 21464 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:44.297698 21464 heartbeater.cc:511] Master 127.1.221.190:34297 requested a full tablet report, sending...
I20260926 03:42:44.300863 21275 ts_manager.cc:194] Registered new tserver with Master: 937e47a2920847d6bf471eb495f03996 (127.1.221.129:46549)
I20260926 03:42:44.302573 21275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:39647
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260926 03:42:44.341105 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:44.711437 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:44.711764 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:44.711874 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:44.720021 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:44.720156 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:44.720293 21470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:44.730948 21470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:44.732385 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:44.733929 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:44.747932 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:44.747896 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:44.747910 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:44.748934 21470 server_base.cc:1034] running on GCE node
I20260926 03:42:44.749573 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:44.750597 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:44.751827 21470 hybrid_clock.cc:648] HybridClock initialized: now 1790394164751763 us; error 47 us; skew 500 ppm
I20260926 03:42:44.752205 21470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:44 dist-test-slave-lqrk krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:44.755509 21470 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260926 03:42:44.757230 21470 webserver.cc:460] Webserver started at http://127.1.221.130:39083/ using document root <none> and password file <none>
I20260926 03:42:44.758038 21470 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:44.758204 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:44.758553 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:44.763609 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "0a272edcfe714b25a8ac82595e6e26ee"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "185842da6fd7737186ced09e28d88ee2"
server_key_iv: "c4e7e70bb057c6094564b14f1c87d9ee"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.764679 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "0a272edcfe714b25a8ac82595e6e26ee"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "185842da6fd7737186ced09e28d88ee2"
server_key_iv: "c4e7e70bb057c6094564b14f1c87d9ee"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.771299 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:42:44.775329 21485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.776899 21470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:44.777076 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "0a272edcfe714b25a8ac82595e6e26ee"
format_stamp: "Formatted at 2026-09-26 03:42:44 on dist-test-slave-lqrk"
server_key: "185842da6fd7737186ced09e28d88ee2"
server_key_iv: "c4e7e70bb057c6094564b14f1c87d9ee"
server_key_version: "encryptionkey@0"
I20260926 03:42:44.777290 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:44.808102 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:44.813133 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:44.813486 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:44.814579 21470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:44.816138 21470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:44.816269 21470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.816416 21470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:44.816486 21470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:44.855311 21470 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36887
I20260926 03:42:44.855314 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36887 every 8 connection(s)
I20260926 03:42:44.856808 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:44.858239 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21470
I20260926 03:42:44.858464 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:44.859015 1910 external_mini_cluster.cc:1428] Setting key 327268f045fd595bace4fab402f2a4c8
Sep 26 03:42:44 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:44.882048 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:44.860158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35031 (local address 127.1.221.190:34297)
0926 03:42:44.860743 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:44.860749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:44.862813 (+ 2064us) server_negotiation.cc:407] Connection header received
0926 03:42:44.864327 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:44.864333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:44.864441 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:44.864563 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:44.866324 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.867360 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.869400 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.870039 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.870171 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:44.875451 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:44.875458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:44.875463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:44.875505 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:44.878246 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:44.879320 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:44.879324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:44.879327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:44.879530 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:44.880055 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:44.880059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:44.880062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:44.880410 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:44.880554 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:44.881489 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 03:42:44.881747 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":286,"threads_started":1}
I20260926 03:42:44.883356 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:44.860851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34297 (local address 127.1.221.130:35031)
0926 03:42:44.861841 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:44.861892 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:44.863984 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:44.864903 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:44.864916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:44.865503 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:44.866124 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:44.866143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.867594 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.867599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:44.869194 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:44.869215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:44.870342 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:44.870351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:44.870752 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:44.871671 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:44.871724 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:44.875061 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:44.878463 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:44.878473 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:44.878494 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:44.879145 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:44.879645 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:44.879650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:44.879653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:44.879912 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:44.880559 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:44.880568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:44.881014 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 03:42:44.881334 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 03:42:44.881898 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":456,"threads_started":1}
I20260926 03:42:44.886109 21598 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34297
I20260926 03:42:44.887030 21598 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260926 03:42:44.888306 21598 heartbeater.cc:511] Master 127.1.221.190:34297 requested a full tablet report, sending...
I20260926 03:42:44.891080 21275 ts_manager.cc:194] Registered new tserver with Master: 0a272edcfe714b25a8ac82595e6e26ee (127.1.221.130:36887)
I20260926 03:42:44.892028 21275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:35031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260926 03:42:44.932788 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:42:45.306550 21464 heartbeater.cc:503] Master 127.1.221.190:34297 was elected leader, sending a full tablet report...
W20260926 03:42:45.309566 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:45.309866 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:45.309993 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:45.318913 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:45.319068 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:45.319218 21604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:45.330439 21604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:45.331852 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:45.333194 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:45.347817 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:45.347877 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:45.348076 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:45.348558 21604 server_base.cc:1034] running on GCE node
I20260926 03:42:45.349208 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:45.350231 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:45.351481 21604 hybrid_clock.cc:648] HybridClock initialized: now 1790394165351435 us; error 34 us; skew 500 ppm
I20260926 03:42:45.351835 21604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:45 dist-test-slave-lqrk krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:45.355176 21604 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260926 03:42:45.356755 21604 webserver.cc:460] Webserver started at http://127.1.221.131:42753/ using document root <none> and password file <none>
I20260926 03:42:45.357471 21604 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:45.357585 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:45.357928 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:45.361112 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "1c58cc437c554755ab632207e5371a71"
format_stamp: "Formatted at 2026-09-26 03:42:45 on dist-test-slave-lqrk"
server_key: "984f6e073cb9572bd72ea8622d2daab7"
server_key_iv: "892e8d545e28e8bd887a18022e9028bf"
server_key_version: "encryptionkey@0"
I20260926 03:42:45.361938 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "1c58cc437c554755ab632207e5371a71"
format_stamp: "Formatted at 2026-09-26 03:42:45 on dist-test-slave-lqrk"
server_key: "984f6e073cb9572bd72ea8622d2daab7"
server_key_iv: "892e8d545e28e8bd887a18022e9028bf"
server_key_version: "encryptionkey@0"
I20260926 03:42:45.368029 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:42:45.372344 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:45.374096 21604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:42:45.374313 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "1c58cc437c554755ab632207e5371a71"
format_stamp: "Formatted at 2026-09-26 03:42:45 on dist-test-slave-lqrk"
server_key: "984f6e073cb9572bd72ea8622d2daab7"
server_key_iv: "892e8d545e28e8bd887a18022e9028bf"
server_key_version: "encryptionkey@0"
I20260926 03:42:45.374548 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:45.398095 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:45.403574 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:45.403874 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:45.404949 21604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:45.406679 21604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:45.406783 21604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:45.406885 21604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:45.406930 21604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:45.442756 21604 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36025
I20260926 03:42:45.442791 21731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36025 every 8 connection(s)
I20260926 03:42:45.444658 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:45.446576 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21604
I20260926 03:42:45.446794 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:45.447407 1910 external_mini_cluster.cc:1428] Setting key b265442d16937d01fd0482480707809d
Sep 26 03:42:45 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:42:45.471451 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.449229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:42555 (local address 127.1.221.190:34297)
0926 03:42:45.450041 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.450052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.451552 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 03:42:45.453640 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.453646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.453749 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.454049 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:45.455909 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.456723 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.459148 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.459642 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.459763 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.464145 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:45.464177 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:45.464184 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:45.464254 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:45.468004 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.468904 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.468909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.468916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.469045 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.469494 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.469498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.469500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.469824 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:45.469982 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.470766 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.471031 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":532,"threads_started":1}
I20260926 03:42:45.472469 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.450349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34297 (local address 127.1.221.131:42555)
0926 03:42:45.451326 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.451434 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.453222 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.454313 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.454323 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.454986 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:45.455696 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.455718 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.457101 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.457106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.458939 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.458960 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.459913 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.459920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.460153 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.460858 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:42:45.460888 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:45.463806 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:45.468206 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:45.468215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:45.468239 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:45.468745 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:45.469160 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:42:45.469163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:42:45.469166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:42:45.469372 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:42:45.469956 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:45.469965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:45.470354 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.470610 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.471242 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":442,"threads_started":1}
I20260926 03:42:45.474510 21732 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34297
I20260926 03:42:45.474951 21732 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:45.475872 21732 heartbeater.cc:511] Master 127.1.221.190:34297 requested a full tablet report, sending...
I20260926 03:42:45.477695 21274 ts_manager.cc:194] Registered new tserver with Master: 1c58cc437c554755ab632207e5371a71 (127.1.221.131:36025)
I20260926 03:42:45.478658 21274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:42555
I20260926 03:42:45.491163 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:45.510272 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.495787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.1.221.190:34297)
0926 03:42:45.495964 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.495969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.496473 (+ 504us) server_negotiation.cc:407] Connection header received
0926 03:42:45.496638 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.496642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.496726 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.496832 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:45.498400 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.499198 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.500733 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.501217 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.501338 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.504016 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:45.504022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:45.504025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:45.504068 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:45.506875 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.507848 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.507853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.507855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.508092 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.508534 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.508538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.508540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.508952 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:45.509154 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.509719 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.509947 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:42:45.515355 21274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:45.524382 21274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:45.554358 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.538346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36025 (local address 127.0.0.1:51344)
0926 03:42:45.539607 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.539666 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.539935 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.542736 (+ 2801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.542746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.542844 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:45.543471 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.543492 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.546961 (+ 3469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.546969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.550241 (+ 3272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.550275 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.553074 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.553082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.553376 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.553448 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.553629 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.553875 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":15116,"threads_started":1}
I20260926 03:42:45.554792 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.538602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.1.221.131:36025)
0926 03:42:45.540671 (+ 2069us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.540696 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.540739 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:42:45.540867 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.540877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.541820 (+ 943us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.542091 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:45.544338 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.546596 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.550675 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.552791 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.552965 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.553250 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.554184 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.554386 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1688,"threads_started":1}
I20260926 03:42:45.559022 21667 tablet_service.cc:1467] Processing CreateTablet for tablet 7375d204c1904e07954c78f0697293f7 (DEFAULT_TABLE table=test-table [id=85d7650dd18745b0892834abaf2ba152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:45.560870 21667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7375d204c1904e07954c78f0697293f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:45.561847 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.539130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.1.221.130:36887)
0926 03:42:45.553001 (+ 13871us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.553011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.553044 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:42:45.553157 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.553167 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.553593 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.553950 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:45.555487 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.556764 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.559201 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.560780 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.560936 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.561256 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.561383 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.561551 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13675,"thread_start_us":13519,"threads_started":1}
I20260926 03:42:45.561849 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.538536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36887 (local address 127.0.0.1:37994)
0926 03:42:45.550640 (+ 12104us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.550667 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.550848 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.554243 (+ 3395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.554249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.554285 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:45.554670 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.554681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.556904 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.556910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.559034 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.559047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.561012 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.561019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.561201 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.561234 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.561337 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.561486 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11910,"thread_start_us":520,"threads_started":1}
I20260926 03:42:45.564399 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.540649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34922)
0926 03:42:45.553820 (+ 13171us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.553841 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.553963 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.556359 (+ 2396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.556365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.556395 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:45.556797 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.556812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.558856 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.558862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.561104 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.561122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.563429 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.563450 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.563635 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.563670 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.563774 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.563940 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13009,"mutex_wait_us":5,"thread_start_us":10300,"threads_started":1}
I20260926 03:42:45.564503 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.541824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.1.221.129:46549)
0926 03:42:45.555488 (+ 13664us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.555498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.555527 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:42:45.555625 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.555633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.555883 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.556089 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:45.557015 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.558657 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.561356 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.563108 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.563340 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.563626 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.563833 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.564018 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13452,"thread_start_us":13273,"threads_started":1}
I20260926 03:42:45.566994 21533 tablet_service.cc:1467] Processing CreateTablet for tablet 7375d204c1904e07954c78f0697293f7 (DEFAULT_TABLE table=test-table [id=85d7650dd18745b0892834abaf2ba152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:45.569216 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7375d204c1904e07954c78f0697293f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:45.573046 21399 tablet_service.cc:1467] Processing CreateTablet for tablet 7375d204c1904e07954c78f0697293f7 (DEFAULT_TABLE table=test-table [id=85d7650dd18745b0892834abaf2ba152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:45.574777 21399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7375d204c1904e07954c78f0697293f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:45.577263 21751 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Bootstrap starting.
I20260926 03:42:45.581738 21751 tablet_bootstrap.cc:654] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:45.582971 21751 log.cc:824] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:45.586477 21753 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Bootstrap starting.
I20260926 03:42:45.586742 21751 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: No bootstrap required, opened a new log
I20260926 03:42:45.587035 21751 ts_tablet_manager.cc:1397] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:42:45.591460 21753 tablet_bootstrap.cc:654] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:45.591601 21754 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Bootstrap starting.
I20260926 03:42:45.593073 21753 log.cc:824] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:45.595487 21754 tablet_bootstrap.cc:654] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:45.596629 21754 log.cc:824] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:45.596987 21753 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: No bootstrap required, opened a new log
I20260926 03:42:45.597281 21753 ts_tablet_manager.cc:1397] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:42:45.598474 21751 raft_consensus.cc:348] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.598958 21751 raft_consensus.cc:374] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:45.599063 21751 raft_consensus.cc:729] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c58cc437c554755ab632207e5371a71, State: Initialized, Role: FOLLOWER
I20260926 03:42:45.599520 21754 tablet_bootstrap.cc:492] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: No bootstrap required, opened a new log
I20260926 03:42:45.599809 21754 ts_tablet_manager.cc:1397] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:42:45.599905 21751 consensus_queue.cc:260] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.608604 21732 heartbeater.cc:503] Master 127.1.221.190:34297 was elected leader, sending a full tablet report...
I20260926 03:42:45.609195 21754 raft_consensus.cc:348] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.609568 21751 ts_tablet_manager.cc:1428] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Time spent starting tablet: real 0.022s user 0.012s sys 0.012s
I20260926 03:42:45.609781 21754 raft_consensus.cc:374] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:45.609894 21754 raft_consensus.cc:729] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937e47a2920847d6bf471eb495f03996, State: Initialized, Role: FOLLOWER
I20260926 03:42:45.610613 21754 consensus_queue.cc:260] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.611260 21753 raft_consensus.cc:348] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.611799 21753 raft_consensus.cc:374] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:45.611896 21753 raft_consensus.cc:729] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a272edcfe714b25a8ac82595e6e26ee, State: Initialized, Role: FOLLOWER
I20260926 03:42:45.612620 21754 ts_tablet_manager.cc:1428] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260926 03:42:45.612653 21753 consensus_queue.cc:260] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.614058 21598 heartbeater.cc:503] Master 127.1.221.190:34297 was elected leader, sending a full tablet report...
I20260926 03:42:45.614758 21753 ts_tablet_manager.cc:1428] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260926 03:42:45.689651 21757 raft_consensus.cc:482] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:45.690025 21757 raft_consensus.cc:504] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.692849 21757 leader_election.cc:288] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a272edcfe714b25a8ac82595e6e26ee (127.1.221.130:36887), 937e47a2920847d6bf471eb495f03996 (127.1.221.129:46549)
W20260926 03:42:45.697649 21733 tablet.cc:2369] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:45.698894 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.693523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54659 (local address 127.1.221.130:36887)
0926 03:42:45.693745 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.693752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.693778 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:42:45.693999 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.694003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.694076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.694169 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:45.694558 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.695409 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.697096 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.697952 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.698115 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.698198 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.698493 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.698627 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:42:45.698894 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.693341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36887 (local address 127.1.221.131:54659)
0926 03:42:45.693685 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.693708 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.693844 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.694199 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.694203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.694231 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:45.694418 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.694428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.695561 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.695565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.696917 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.696931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.698116 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.698123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.698336 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.698363 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.698449 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.698601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260926 03:42:45.699985 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7375d204c1904e07954c78f0697293f7" candidate_uuid: "1c58cc437c554755ab632207e5371a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a272edcfe714b25a8ac82595e6e26ee" is_pre_election: true
I20260926 03:42:45.700371 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.693652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46807 (local address 127.1.221.129:46549)
0926 03:42:45.693851 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.693857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.694524 (+ 667us) server_negotiation.cc:407] Connection header received
0926 03:42:45.694697 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.694701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.694764 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.694841 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:45.695407 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.696101 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.698119 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.699213 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.699408 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.699478 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.699826 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.699981 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:42:45.700414 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.693445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.1.221.131:46807)
0926 03:42:45.694423 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.694438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:45.694566 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:45.694879 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:45.694883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:45.694906 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:45.695221 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.695230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.696263 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.696266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.697878 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:45.697894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.699454 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.699462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:45.699640 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.699666 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:42:45.699765 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:42:45.699905 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":583,"threads_started":1}
I20260926 03:42:45.700546 21553 raft_consensus.cc:2454] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c58cc437c554755ab632207e5371a71 in term 0.
I20260926 03:42:45.701395 21622 leader_election.cc:302] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a272edcfe714b25a8ac82595e6e26ee, 1c58cc437c554755ab632207e5371a71; no voters:
I20260926 03:42:45.701784 21757 raft_consensus.cc:2781] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:45.701911 21757 raft_consensus.cc:482] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:42:45.701983 21757 raft_consensus.cc:3037] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:45.701989 21419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7375d204c1904e07954c78f0697293f7" candidate_uuid: "1c58cc437c554755ab632207e5371a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937e47a2920847d6bf471eb495f03996" is_pre_election: true
I20260926 03:42:45.703177 21419 raft_consensus.cc:2454] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c58cc437c554755ab632207e5371a71 in term 0.
I20260926 03:42:45.705557 21757 raft_consensus.cc:504] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.706691 21757 leader_election.cc:288] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [CANDIDATE]: Term 1 election: Requested vote from peers 0a272edcfe714b25a8ac82595e6e26ee (127.1.221.130:36887), 937e47a2920847d6bf471eb495f03996 (127.1.221.129:46549)
I20260926 03:42:45.707001 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7375d204c1904e07954c78f0697293f7" candidate_uuid: "1c58cc437c554755ab632207e5371a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a272edcfe714b25a8ac82595e6e26ee"
I20260926 03:42:45.707098 21419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7375d204c1904e07954c78f0697293f7" candidate_uuid: "1c58cc437c554755ab632207e5371a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937e47a2920847d6bf471eb495f03996"
I20260926 03:42:45.707230 21553 raft_consensus.cc:3037] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:45.707353 21419 raft_consensus.cc:3037] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:45.709951 21553 raft_consensus.cc:2454] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c58cc437c554755ab632207e5371a71 in term 1.
I20260926 03:42:45.709944 21419 raft_consensus.cc:2454] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c58cc437c554755ab632207e5371a71 in term 1.
I20260926 03:42:45.710500 21622 leader_election.cc:302] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a272edcfe714b25a8ac82595e6e26ee, 1c58cc437c554755ab632207e5371a71; no voters:
I20260926 03:42:45.710855 21757 raft_consensus.cc:2781] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:45.711416 21757 raft_consensus.cc:686] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 1 LEADER]: Becoming Leader. State: Replica: 1c58cc437c554755ab632207e5371a71, State: Running, Role: LEADER
I20260926 03:42:45.711952 21757 consensus_queue.cc:237] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } }
I20260926 03:42:45.717341 21274 catalog_manager.cc:5495] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c58cc437c554755ab632207e5371a71 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "1c58cc437c554755ab632207e5371a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c58cc437c554755ab632207e5371a71" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 } health_report { overall_health: UNKNOWN } } }
I20260926 03:42:45.758021 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.752370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.1.221.131:36025)
0926 03:42:45.752589 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.752594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.752657 (+ 63us) server_negotiation.cc:407] Connection header received
0926 03:42:45.752900 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.752904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.752998 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.753139 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:45.753523 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.754282 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.755742 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.756153 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.756265 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.756807 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:45.757411 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:45.757540 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.757651 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.757749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:42:45.768421 21553 raft_consensus.cc:1261] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 1 FOLLOWER]: Refusing update from remote peer 1c58cc437c554755ab632207e5371a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:45.768421 21419 raft_consensus.cc:1261] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 1 FOLLOWER]: Refusing update from remote peer 1c58cc437c554755ab632207e5371a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:45.769596 21761 consensus_queue.cc:1035] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "937e47a2920847d6bf471eb495f03996" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:45.770138 21757 consensus_queue.cc:1035] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a272edcfe714b25a8ac82595e6e26ee" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:42:45.770874 21465 tablet.cc:2369] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:45.782899 21764 mvcc.cc:205] Tried to move back new op lower bound from 7333454502975082496 to 7333454502760751104. Current Snapshot: MvccSnapshot[applied={T|T < 7333454502975082496}]
I20260926 03:42:45.799480 21274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:45.799780 21274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:45.804754 21274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5f9e133459f47bb84558090284ef7ff: Sending DeleteTablet for 3 replicas of tablet 7375d204c1904e07954c78f0697293f7
I20260926 03:42:45.806131 21667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7375d204c1904e07954c78f0697293f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:45 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:51344
I20260926 03:42:45.806254 21399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7375d204c1904e07954c78f0697293f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:45 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34922
I20260926 03:42:45.806324 21533 tablet_service.cc:1514] Processing DeleteTablet for tablet 7375d204c1904e07954c78f0697293f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:45 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:37994
I20260926 03:42:45.808697 21398 tablet_service.cc:1514] Processing DeleteTablet for tablet 7375d204c1904e07954c78f0697293f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:45 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34922
I20260926 03:42:45.811570 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21336
I20260926 03:42:45.825448 21775 tablet_replica.cc:330] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: stopping tablet replica
I20260926 03:42:45.825448 21774 tablet_replica.cc:330] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: stopping tablet replica
I20260926 03:42:45.825986 21774 raft_consensus.cc:2229] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:45.825976 21775 raft_consensus.cc:2229] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:45.826562 21775 raft_consensus.cc:2258] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:45.826669 21774 raft_consensus.cc:2258] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:45.828012 21776 tablet_replica.cc:330] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: stopping tablet replica
I20260926 03:42:45.828450 21776 raft_consensus.cc:2229] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:45.828976 21776 raft_consensus.cc:2258] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:45.830256 21774 ts_tablet_manager.cc:1905] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:45.830840 21775 ts_tablet_manager.cc:1905] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:42:45 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260926 03:42:45.837952 21774 ts_tablet_manager.cc:1918] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:45.838155 21774 log.cc:1196] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/7375d204c1904e07954c78f0697293f7
I20260926 03:42:45.838688 21774 ts_tablet_manager.cc:1939] T 7375d204c1904e07954c78f0697293f7 P 1c58cc437c554755ab632207e5371a71: Deleting consensus metadata
I20260926 03:42:45.839814 21776 ts_tablet_manager.cc:1905] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:45.840713 21260 catalog_manager.cc:4841] TS 1c58cc437c554755ab632207e5371a71 (127.1.221.131:36025): tablet 7375d204c1904e07954c78f0697293f7 (table test-table [id=85d7650dd18745b0892834abaf2ba152]) successfully deleted
I20260926 03:42:45.845592 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.813560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.1.221.129:46549)
0926 03:42:45.813761 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.813766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.820132 (+ 6366us) server_negotiation.cc:407] Connection header received
0926 03:42:45.821218 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.821226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.821329 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.821527 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:45.827520 (+ 5993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.828668 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.831068 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.831804 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.831917 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.837991 (+ 6074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:45.838008 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:45.838031 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:45.838089 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:45.842110 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.843078 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.843098 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.843102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.843287 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.843789 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.843794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.843798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.844307 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:45.844656 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:45.844951 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:42:45.845191 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:42:45.846166 21776 ts_tablet_manager.cc:1918] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:45.846342 21459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:45.846359 21776 log.cc:1196] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/7375d204c1904e07954c78f0697293f7
I20260926 03:42:45.846916 21776 ts_tablet_manager.cc:1939] T 7375d204c1904e07954c78f0697293f7 P 0a272edcfe714b25a8ac82595e6e26ee: Deleting consensus metadata
I20260926 03:42:45.850509 21261 catalog_manager.cc:4841] TS 0a272edcfe714b25a8ac82595e6e26ee (127.1.221.130:36887): tablet 7375d204c1904e07954c78f0697293f7 (table test-table [id=85d7650dd18745b0892834abaf2ba152]) successfully deleted
I20260926 03:42:45.951848 21775 ts_tablet_manager.cc:1918] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:45.952126 21775 log.cc:1196] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/7375d204c1904e07954c78f0697293f7
I20260926 03:42:45.952728 21775 ts_tablet_manager.cc:1939] T 7375d204c1904e07954c78f0697293f7 P 937e47a2920847d6bf471eb495f03996: Deleting consensus metadata
I20260926 03:42:45.952908 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21336
I20260926 03:42:45.977339 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21470
W20260926 03:42:45.977169 21262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:46549 (error 108)
I20260926 03:42:45.980443 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.980014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34930)
0926 03:42:45.980178 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:45.980283 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:42:45.980808 21262 catalog_manager.cc:4584] TS 937e47a2920847d6bf471eb495f03996 (127.1.221.129:46549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7375d204c1904e07954c78f0697293f7: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Sep 26 03:42:45 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260926 03:42:46.001660 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:45.978288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.1.221.130:36887)
0926 03:42:45.978500 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:45.978506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:45.978663 (+ 157us) server_negotiation.cc:407] Connection header received
0926 03:42:45.978889 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:45.978895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:45.979003 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:45.979125 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:45.981212 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.982081 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.984966 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:45.985423 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:45.985544 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:45.992803 (+ 7259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:45.992825 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:45.992843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:45.992892 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:45.996687 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.998074 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.998096 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.998099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.998416 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:45.999140 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:45.999147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:45.999150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:45.999842 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:46.000195 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:46.000922 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 03:42:46.001339 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:42:46.003726 21593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:46.026073 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.025185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34946)
0926 03:42:46.025566 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.025718 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 03:42:46.027505 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.027024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34958)
0926 03:42:46.027194 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.027309 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:42:46.035632 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.034786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34964)
0926 03:42:46.035257 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.035420 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
I20260926 03:42:46.036933 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.036482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34978)
0926 03:42:46.036653 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.036766 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:42:46.078364 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.077398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34992)
0926 03:42:46.077841 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.078120 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260926 03:42:46.081180 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.079644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:34996)
0926 03:42:46.080127 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.080297 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260926 03:42:46.114916 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21470
I20260926 03:42:46.116016 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.114561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35006)
0926 03:42:46.114906 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.115011 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 03:42:46.117997 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.117234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35016)
0926 03:42:46.117641 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.117748 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 03:42:46.132009 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21604
Sep 26 03:42:46 dist-test-slave-lqrk krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260926 03:42:46.152367 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.132935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.1.221.131:36025)
0926 03:42:46.133164 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:46.133170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:46.133194 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:42:46.133373 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:46.133385 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:46.133470 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:46.133605 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:46.134969 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:46.136316 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:46.137699 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:46.138395 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:46.138555 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:46.143625 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:46.143639 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:46.143657 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:46.143709 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:46.148210 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:46.149433 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:46.149442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:46.149444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:46.149631 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:46.150133 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:46.150137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:46.150139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:46.150569 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:46.150828 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:46.151798 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 03:42:46.152028 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:42:46.153913 21727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:46.182154 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.181634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35024)
0926 03:42:46.181883 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.181979 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 03:42:46.183259 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.182870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35034)
0926 03:42:46.183065 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.183141 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:42:46.220270 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.219298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35050)
0926 03:42:46.219817 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.220003 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260926 03:42:46.221781 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.221317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46549 (local address 127.0.0.1:35052)
0926 03:42:46.221541 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:46.221644 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:42:46.290460 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21604
I20260926 03:42:46.308223 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21243
I20260926 03:42:46.309567 21305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:46.432575 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21243
2026-09-26T03:42:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T03:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:46Z Disabled control of system clock
I20260926 03:42:46.514972 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42505
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:46.877405 21786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:46.877722 21786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:46.877810 21786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:46.885505 21786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:46.885634 21786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:46.885673 21786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:46.885756 21786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:46.885813 21786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:46.895780 21786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:46.897136 21786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:46.899178 21786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:46.911422 21791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:46.911585 21792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:46.911722 21794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:46.912232 21786 server_base.cc:1034] running on GCE node
I20260926 03:42:46.913081 21786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:46.914898 21786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:46.916181 21786 hybrid_clock.cc:648] HybridClock initialized: now 1790394166916124 us; error 46 us; skew 500 ppm
I20260926 03:42:46.916592 21786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:46.926298 21786 webserver.cc:460] Webserver started at http://127.1.221.190:45093/ using document root <none> and password file <none>
I20260926 03:42:46.927088 21786 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:46.927203 21786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:46.927587 21786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:46.930702 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "7f878648edf74343b7a59559c1097044"
format_stamp: "Formatted at 2026-09-26 03:42:46 on dist-test-slave-lqrk"
server_key: "d60636e3beaa767692b8f8d01ceac963"
server_key_iv: "1ef71027c21aedbca8b9cde7b8a43c50"
server_key_version: "encryptionkey@0"
I20260926 03:42:46.931667 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "7f878648edf74343b7a59559c1097044"
format_stamp: "Formatted at 2026-09-26 03:42:46 on dist-test-slave-lqrk"
server_key: "d60636e3beaa767692b8f8d01ceac963"
server_key_iv: "1ef71027c21aedbca8b9cde7b8a43c50"
server_key_version: "encryptionkey@0"
I20260926 03:42:46.937232 21786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:42:46.941015 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:46.942778 21786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:42:46.942996 21786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "7f878648edf74343b7a59559c1097044"
format_stamp: "Formatted at 2026-09-26 03:42:46 on dist-test-slave-lqrk"
server_key: "d60636e3beaa767692b8f8d01ceac963"
server_key_iv: "1ef71027c21aedbca8b9cde7b8a43c50"
server_key_version: "encryptionkey@0"
I20260926 03:42:46.943238 21786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:46.961429 21786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:46.962576 21786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:46.962968 21786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:46.983934 21786 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42505
I20260926 03:42:46.983943 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42505 every 8 connection(s)
I20260926 03:42:46.985433 21786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:46.989534 21852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:46.992533 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21786
I20260926 03:42:46.992734 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:46.993294 1910 external_mini_cluster.cc:1428] Setting key fc2c1cc994805c5cb892d2fa36c0e349
I20260926 03:42:47.002920 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: Bootstrap starting.
I20260926 03:42:47.006649 21852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:47.007897 21852 log.cc:824] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:47.008942 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:46.995671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.1.221.190:42505)
0926 03:42:46.996815 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:46.996827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:46.996882 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:42:46.998758 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:46.998795 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:46.999436 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:46.999911 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:47.000555 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.001890 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.003308 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.004115 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.004294 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:47.006147 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:47.006157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:47.006174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:47.006783 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:47.006919 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:47.006946 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:47.007075 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:47.007335 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 03:42:47.007720 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":407,"threads_started":1}
I20260926 03:42:47.011162 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: No bootstrap required, opened a new log
I20260926 03:42:47.019542 21852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } }
I20260926 03:42:47.019987 21852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:47.020080 21852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f878648edf74343b7a59559c1097044, State: Initialized, Role: FOLLOWER
I20260926 03:42:47.020740 21852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } }
I20260926 03:42:47.020999 21852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:47.021100 21852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:47.021227 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:47.023615 21852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } }
I20260926 03:42:47.024240 21852 leader_election.cc:302] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f878648edf74343b7a59559c1097044; no voters:
I20260926 03:42:47.024974 21852 leader_election.cc:288] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:47.025113 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:47.026949 21852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:47.029289 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [term 1 LEADER]: Becoming Leader. State: Replica: 7f878648edf74343b7a59559c1097044, State: Running, Role: LEADER
I20260926 03:42:47.029883 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } }
I20260926 03:42:47.033572 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f878648edf74343b7a59559c1097044. Latest consensus state: current_term: 1 leader_uuid: "7f878648edf74343b7a59559c1097044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } } }
I20260926 03:42:47.033595 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f878648edf74343b7a59559c1097044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f878648edf74343b7a59559c1097044" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42505 } } }
I20260926 03:42:47.034076 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:47.034082 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044 [sys.catalog]: This master's current role is: LEADER
W20260926 03:42:47.037956 21871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:42:47.038081 21871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:42:47.038239 21872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:47.040235 21872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:47.050949 21872 catalog_manager.cc:1347] Generated new cluster ID: 5ec948d02af44d398238e56e84de98e6
I20260926 03:42:47.051095 21872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:47.067152 21872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:47.069185 21872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:47.086056 21872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: Generated new TSK 0
I20260926 03:42:47.087317 21872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:47.098826 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--builtin_ntp_servers=127.1.221.148:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:42:47.487739 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:47.488054 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:47.488184 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:47.495797 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:47.495918 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:47.496069 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260926 03:42:47.506126 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:47.507514 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:47.509413 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:47.523489 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:47.523504 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:47.523823 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:47.524364 21876 server_base.cc:1034] running on GCE node
I20260926 03:42:47.525074 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:47.526135 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:47.527486 21876 hybrid_clock.cc:648] HybridClock initialized: now 1790394167527412 us; error 57 us; skew 500 ppm
I20260926 03:42:47.527848 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:47.530478 21876 webserver.cc:460] Webserver started at http://127.1.221.129:44189/ using document root <none> and password file <none>
I20260926 03:42:47.531297 21876 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:47.531427 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:47.531752 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:47.534879 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data/instance:
uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb"
format_stamp: "Formatted at 2026-09-26 03:42:47 on dist-test-slave-lqrk"
server_key: "1c28115b704309840592d55ce9b1da20"
server_key_iv: "5f5fb7e9da051a27f32f4c3cf5ab9792"
server_key_version: "encryptionkey@0"
I20260926 03:42:47.535756 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb"
format_stamp: "Formatted at 2026-09-26 03:42:47 on dist-test-slave-lqrk"
server_key: "1c28115b704309840592d55ce9b1da20"
server_key_iv: "5f5fb7e9da051a27f32f4c3cf5ab9792"
server_key_version: "encryptionkey@0"
I20260926 03:42:47.541314 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:42:47.545135 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:47.546588 21876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:42:47.546832 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal
uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb"
format_stamp: "Formatted at 2026-09-26 03:42:47 on dist-test-slave-lqrk"
server_key: "1c28115b704309840592d55ce9b1da20"
server_key_iv: "5f5fb7e9da051a27f32f4c3cf5ab9792"
server_key_version: "encryptionkey@0"
I20260926 03:42:47.547109 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:47.565428 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:47.566428 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:47.566707 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:47.567853 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:47.569410 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:47.569511 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:47.569591 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:47.569644 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:47.605120 21876 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:42883
I20260926 03:42:47.605150 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:42883 every 8 connection(s)
I20260926 03:42:47.606449 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/data/info.pb
I20260926 03:42:47.608076 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 21876
I20260926 03:42:47.608266 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal/instance
I20260926 03:42:47.609015 1910 external_mini_cluster.cc:1428] Setting key 36023b715a6923ae2fb8ff76c39bf00a
I20260926 03:42:47.611958 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--builtin_ntp_servers=127.1.221.148:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:47.622051 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:47.609636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33805 (local address 127.1.221.190:42505)
0926 03:42:47.610104 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:47.610110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:47.611613 (+ 1503us) server_negotiation.cc:407] Connection header received
0926 03:42:47.614032 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:47.614038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:47.614126 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:47.614387 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:47.615684 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.616636 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.618547 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.619138 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.619275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:47.621116 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:47.621122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:47.621127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:47.621153 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:47.621203 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:47.621218 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:47.621325 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:47.621582 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:42:47.621692 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
I20260926 03:42:47.623196 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:47.610270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42505 (local address 127.1.221.129:33805)
0926 03:42:47.611437 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:47.611495 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:47.613678 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:47.614641 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:47.614651 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:47.614758 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:47.615475 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:47.615503 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.616917 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.616921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:47.618298 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:47.618320 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:47.619445 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:47.619454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:47.619775 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:47.620589 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:47.620592 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:47.620638 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:47.620878 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:47.621353 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:47.621363 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:47.621371 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:42:47.621546 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 03:42:47.621958 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":352,"threads_started":1}
I20260926 03:42:47.624892 22003 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42505
I20260926 03:42:47.625238 22003 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:47.626168 22003 heartbeater.cc:511] Master 127.1.221.190:42505 requested a full tablet report, sending...
I20260926 03:42:47.628630 21817 ts_manager.cc:194] Registered new tserver with Master: d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb (127.1.221.129:42883)
I20260926 03:42:47.630218 21817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:33805
W20260926 03:42:47.970992 22007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:47.971287 22007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:47.971410 22007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:47.979045 22007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:47.979166 22007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:47.979313 22007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260926 03:42:47.992424 22007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:47.994088 22007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:47.996155 22007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:48.010864 22012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:48.010842 22013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:48.010842 22015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:48.012322 22007 server_base.cc:1034] running on GCE node
I20260926 03:42:48.012960 22007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:48.013952 22007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:48.015178 22007 hybrid_clock.cc:648] HybridClock initialized: now 1790394168015122 us; error 51 us; skew 500 ppm
I20260926 03:42:48.015561 22007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:48.018002 22007 webserver.cc:460] Webserver started at http://127.1.221.130:37139/ using document root <none> and password file <none>
I20260926 03:42:48.018705 22007 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:48.018807 22007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:48.019124 22007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:48.022217 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data/instance:
uuid: "a49da3317dc34fca8acceea4f877ebb4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "05354a8945497b8af654850b25f5c210"
server_key_iv: "002bffc4228aaa1e5845bb8cc272fdcf"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.023077 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "a49da3317dc34fca8acceea4f877ebb4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "05354a8945497b8af654850b25f5c210"
server_key_iv: "002bffc4228aaa1e5845bb8cc272fdcf"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.029227 22007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:42:48.033401 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.034982 22007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:42:48.035174 22007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal
uuid: "a49da3317dc34fca8acceea4f877ebb4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "05354a8945497b8af654850b25f5c210"
server_key_iv: "002bffc4228aaa1e5845bb8cc272fdcf"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.035455 22007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:48.056124 22007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:48.056993 22007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:48.057289 22007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:48.058298 22007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:48.059957 22007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:48.060060 22007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.060161 22007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:48.060228 22007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.093171 22007 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:40135
I20260926 03:42:48.093245 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:40135 every 8 connection(s)
I20260926 03:42:48.094437 22007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/data/info.pb
I20260926 03:42:48.101689 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22007
I20260926 03:42:48.101881 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal/instance
I20260926 03:42:48.102280 1910 external_mini_cluster.cc:1428] Setting key 2f1f60a36f6351a0dc7eaf210fdfe83a
I20260926 03:42:48.105331 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--builtin_ntp_servers=127.1.221.148:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:42:48.107069 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.097017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41691 (local address 127.1.221.190:42505)
0926 03:42:48.097209 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.097215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.098517 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 03:42:48.100382 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.100388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.100476 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.100562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:48.101605 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.102410 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.104216 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.104812 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.105007 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.106309 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:48.106316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:48.106320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:48.106342 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:48.106382 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:48.106398 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:48.106487 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.106672 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.106771 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:42:48.108099 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.097388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42505 (local address 127.1.221.130:41691)
0926 03:42:48.098343 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.098379 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.100106 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.100798 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.100807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.100895 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:48.101443 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.101459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.102551 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.102555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.104070 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.104087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.105080 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.105085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.105295 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.105956 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:48.105959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:48.106005 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:48.106102 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:48.106491 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:48.106498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:48.106504 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.106645 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.106996 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":271,"threads_started":1}
I20260926 03:42:48.109588 22134 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42505
I20260926 03:42:48.109930 22134 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:48.110827 22134 heartbeater.cc:511] Master 127.1.221.190:42505 requested a full tablet report, sending...
I20260926 03:42:48.112463 21817 ts_manager.cc:194] Registered new tserver with Master: a49da3317dc34fca8acceea4f877ebb4 (127.1.221.130:40135)
I20260926 03:42:48.113490 21817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:41691
W20260926 03:42:48.455657 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:48.455950 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:48.456058 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:48.463649 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:48.463762 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:48.463904 22137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260926 03:42:48.473188 22137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:48.474510 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:48.476334 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:48.490000 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:48.489960 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:48.489921 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:48.491015 22137 server_base.cc:1034] running on GCE node
I20260926 03:42:48.491694 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:48.492707 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:48.493943 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790394168493879 us; error 42 us; skew 500 ppm
I20260926 03:42:48.494282 22137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:42:48.496572 22137 webserver.cc:460] Webserver started at http://127.1.221.131:35859/ using document root <none> and password file <none>
I20260926 03:42:48.497215 22137 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:48.497296 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:48.497560 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:48.500442 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data/instance:
uuid: "20e0089b04594ec49a326ada479400f4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "de371390533dbeffbd6050f8346e118d"
server_key_iv: "797801703fac7aad9182b81e87681cce"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.501175 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "20e0089b04594ec49a326ada479400f4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "de371390533dbeffbd6050f8346e118d"
server_key_iv: "797801703fac7aad9182b81e87681cce"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.506464 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:42:48.510406 22151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.511930 22137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:42:48.512110 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal
uuid: "20e0089b04594ec49a326ada479400f4"
format_stamp: "Formatted at 2026-09-26 03:42:48 on dist-test-slave-lqrk"
server_key: "de371390533dbeffbd6050f8346e118d"
server_key_iv: "797801703fac7aad9182b81e87681cce"
server_key_version: "encryptionkey@0"
I20260926 03:42:48.512333 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:48.546042 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:48.546967 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:48.547295 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:48.548386 22137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:42:48.549916 22137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:42:48.550026 22137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.550153 22137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:42:48.550223 22137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:48.584993 22137 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32945
I20260926 03:42:48.585057 22263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32945 every 8 connection(s)
I20260926 03:42:48.586624 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/data/info.pb
I20260926 03:42:48.593130 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22137
I20260926 03:42:48.593299 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal/instance
I20260926 03:42:48.594029 1910 external_mini_cluster.cc:1428] Setting key f41d39ba791794d5974a7ad21e443ba7
I20260926 03:42:48.600562 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.589337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:47855 (local address 127.1.221.190:42505)
0926 03:42:48.589629 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.589635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.591298 (+ 1663us) server_negotiation.cc:407] Connection header received
0926 03:42:48.593144 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.593151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.593235 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.593322 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:48.594935 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.595873 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.597483 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.598079 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.598213 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.599705 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:48.599711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:48.599714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:48.599734 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:48.599772 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:48.599784 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:48.599862 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.600103 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.600207 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 03:42:48.601780 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.590141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42505 (local address 127.1.221.131:47855)
0926 03:42:48.591131 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.591168 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.592811 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.593629 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.593642 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.593880 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:42:48.594726 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.594749 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.596035 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.596039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.597298 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.597312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.598364 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.598371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.598621 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.599305 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:42:48.599309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:42:48.599367 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:42:48.599500 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:42:48.599881 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:42:48.599892 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:42:48.599900 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.600076 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.600526 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":307,"threads_started":1}
I20260926 03:42:48.603356 22264 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42505
I20260926 03:42:48.603685 22264 heartbeater.cc:464] Registering TS with master...
I20260926 03:42:48.604446 22264 heartbeater.cc:511] Master 127.1.221.190:42505 requested a full tablet report, sending...
I20260926 03:42:48.606086 21817 ts_manager.cc:194] Registered new tserver with Master: 20e0089b04594ec49a326ada479400f4 (127.1.221.131:32945)
I20260926 03:42:48.607239 21817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:47855
I20260926 03:42:48.610702 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:42:48.620880 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.615132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.1.221.190:42505)
0926 03:42:48.615342 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.615349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.615731 (+ 382us) server_negotiation.cc:407] Connection header received
0926 03:42:48.615855 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.615859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.615952 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.616046 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:48.616569 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.617272 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.618638 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.619098 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.619236 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.620322 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:48.620326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:48.620329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:48.620350 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:48.620382 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:48.620394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:48.620463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.620553 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.620633 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:42:48.625118 21817 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:42:48.632709 21817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:42:48.633548 22003 heartbeater.cc:503] Master 127.1.221.190:42505 was elected leader, sending a full tablet report...
I20260926 03:42:48.699222 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.645799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.1.221.131:32945)
0926 03:42:48.688239 (+ 42440us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.688247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.688279 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:42:48.688359 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.688364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.688538 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.688705 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:48.690092 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.691933 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.695531 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.697093 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.697290 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.698483 (+ 1193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.698598 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.698798 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42269,"thread_start_us":42103,"threads_started":1}
I20260926 03:42:48.699426 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.646443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.1.221.130:40135)
0926 03:42:48.680719 (+ 34276us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.680731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.685783 (+ 5052us) server_negotiation.cc:407] Connection header received
0926 03:42:48.686687 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.686698 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.687155 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.687421 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:48.690057 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.691888 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.695858 (+ 3970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.697222 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.697351 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.697576 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.698910 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.699101 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34002,"thread_start_us":35981,"threads_started":1}
I20260926 03:42:48.700139 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.645506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40135 (local address 127.0.0.1:39638)
0926 03:42:48.685607 (+ 40101us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.685630 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.686369 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.687617 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.687626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.687672 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:48.689361 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.689379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.692081 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.692086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.695472 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.695510 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.697788 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.697796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.698212 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.698253 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.698337 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.699667 (+ 1330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39936,"mutex_wait_us":96,"thread_start_us":39780,"threads_started":1}
I20260926 03:42:48.700939 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.645382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32945 (local address 127.0.0.1:35962)
0926 03:42:48.683693 (+ 38311us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.683760 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.684016 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.688723 (+ 4707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.688729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.688754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:48.689501 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.689517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.692118 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.692125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.695252 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.695269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.697399 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.697408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.697702 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.697783 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.698135 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.700449 (+ 2314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37894,"mutex_wait_us":1969,"thread_start_us":44370,"threads_started":1}
I20260926 03:42:48.701484 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.645923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:42883 (local address 127.0.0.1:33820)
0926 03:42:48.690026 (+ 44103us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.690046 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.690396 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.690978 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.690983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.691007 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:48.691317 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.691327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.693306 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.693311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.695886 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.695899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.698712 (+ 2813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.698717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.698966 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.700447 (+ 1481us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.700561 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.700996 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43981,"mutex_wait_us":1478,"thread_start_us":38447,"threads_started":1}
I20260926 03:42:48.702706 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.646866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.1.221.129:42883)
0926 03:42:48.681799 (+ 34933us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.681819 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.690169 (+ 8350us) server_negotiation.cc:407] Connection header received
0926 03:42:48.690586 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.690594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.690793 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.690965 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:48.691477 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.693101 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.696147 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.697890 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.698242 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.698578 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.702078 (+ 3500us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.702307 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34512,"thread_start_us":35003,"threads_started":1}
I20260926 03:42:48.705272 22199 tablet_service.cc:1467] Processing CreateTablet for tablet 8505a20f56be4fbbbd236f428079ce98 (DEFAULT_TABLE table=test-table [id=d7bf180a0dd84cf089e0e37255118b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:48.705308 22069 tablet_service.cc:1467] Processing CreateTablet for tablet 8505a20f56be4fbbbd236f428079ce98 (DEFAULT_TABLE table=test-table [id=d7bf180a0dd84cf089e0e37255118b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:48.707573 22199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8505a20f56be4fbbbd236f428079ce98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:48.707872 22069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8505a20f56be4fbbbd236f428079ce98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:48.708294 21938 tablet_service.cc:1467] Processing CreateTablet for tablet 8505a20f56be4fbbbd236f428079ce98 (DEFAULT_TABLE table=test-table [id=d7bf180a0dd84cf089e0e37255118b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:42:48.711098 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8505a20f56be4fbbbd236f428079ce98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:48.723562 22282 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Bootstrap starting.
I20260926 03:42:48.727630 22282 tablet_bootstrap.cc:654] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:48.729846 22282 log.cc:824] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:48.734139 22282 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: No bootstrap required, opened a new log
I20260926 03:42:48.734458 22282 ts_tablet_manager.cc:1397] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 03:42:48.735716 22283 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Bootstrap starting.
I20260926 03:42:48.739331 22284 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Bootstrap starting.
I20260926 03:42:48.739907 22283 tablet_bootstrap.cc:654] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:48.741782 22283 log.cc:824] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:48.743105 22282 raft_consensus.cc:348] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.743637 22282 raft_consensus.cc:374] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:48.743750 22282 raft_consensus.cc:729] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e0089b04594ec49a326ada479400f4, State: Initialized, Role: FOLLOWER
I20260926 03:42:48.743916 22284 tablet_bootstrap.cc:654] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:48.744678 22283 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: No bootstrap required, opened a new log
I20260926 03:42:48.744537 22282 consensus_queue.cc:260] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.744951 22283 ts_tablet_manager.cc:1397] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260926 03:42:48.745199 22284 log.cc:824] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:48.745810 22264 heartbeater.cc:503] Master 127.1.221.190:42505 was elected leader, sending a full tablet report...
I20260926 03:42:48.746299 22282 ts_tablet_manager.cc:1428] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 03:42:48.748387 22284 tablet_bootstrap.cc:492] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: No bootstrap required, opened a new log
I20260926 03:42:48.748647 22284 ts_tablet_manager.cc:1397] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260926 03:42:48.756094 22283 raft_consensus.cc:348] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.756497 22283 raft_consensus.cc:374] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:48.756595 22283 raft_consensus.cc:729] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49da3317dc34fca8acceea4f877ebb4, State: Initialized, Role: FOLLOWER
I20260926 03:42:48.757079 22284 raft_consensus.cc:348] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.757238 22283 consensus_queue.cc:260] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.757484 22284 raft_consensus.cc:374] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:48.757579 22284 raft_consensus.cc:729] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb, State: Initialized, Role: FOLLOWER
I20260926 03:42:48.758224 22284 consensus_queue.cc:260] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.758538 22134 heartbeater.cc:503] Master 127.1.221.190:42505 was elected leader, sending a full tablet report...
I20260926 03:42:48.759177 22283 ts_tablet_manager.cc:1428] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 03:42:48.760015 22284 ts_tablet_manager.cc:1428] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
W20260926 03:42:48.838477 22265 tablet.cc:2369] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:42:48.846539 22135 tablet.cc:2369] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:42:48.860445 22004 tablet.cc:2369] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:42:48.974193 22287 raft_consensus.cc:482] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:42:48.974493 22287 raft_consensus.cc:504] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.976261 22287 leader_election.cc:288] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a49da3317dc34fca8acceea4f877ebb4 (127.1.221.130:40135), d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb (127.1.221.129:42883)
I20260926 03:42:48.981845 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.976828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40135 (local address 127.1.221.131:44383)
0926 03:42:48.976993 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.977017 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.977133 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.977513 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.977518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.977547 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:48.977723 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.977731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.978726 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.978730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.980073 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.980085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.981132 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.981137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.981292 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.981319 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.981398 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.981539 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:42:48.981834 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.976942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44383 (local address 127.1.221.130:40135)
0926 03:42:48.977093 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.977097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.977114 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:42:48.977248 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.977252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.977344 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.977438 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:48.977883 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.978568 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.980231 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.980976 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.981139 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.981215 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.981443 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.981567 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:42:48.982851 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.976851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:42883 (local address 127.1.221.131:54429)
0926 03:42:48.977921 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0926 03:42:48.977934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:42:48.978045 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:42:48.978357 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:42:48.978361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:42:48.978383 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:42:48.978684 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.978693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.979699 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.979702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.981007 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:42:48.981018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.982242 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.982248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:42:48.982408 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.982441 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:42:48.982516 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:42:48.982612 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":358,"thread_start_us":381,"threads_started":1}
I20260926 03:42:48.982956 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:48.976721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54429 (local address 127.1.221.129:42883)
0926 03:42:48.976950 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:48.976956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:48.978024 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 03:42:48.978172 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:48.978176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:48.978245 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:48.978322 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:42:48.978833 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.979548 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.981185 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:48.981987 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:48.982140 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:48.982201 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:48.982575 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 03:42:48.982703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:42:48.983002 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8505a20f56be4fbbbd236f428079ce98" candidate_uuid: "20e0089b04594ec49a326ada479400f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49da3317dc34fca8acceea4f877ebb4" is_pre_election: true
I20260926 03:42:48.983675 22089 raft_consensus.cc:2454] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e0089b04594ec49a326ada479400f4 in term 0.
I20260926 03:42:48.983997 21958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8505a20f56be4fbbbd236f428079ce98" candidate_uuid: "20e0089b04594ec49a326ada479400f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" is_pre_election: true
I20260926 03:42:48.984472 21958 raft_consensus.cc:2454] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e0089b04594ec49a326ada479400f4 in term 0.
I20260926 03:42:48.984458 22154 leader_election.cc:302] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e0089b04594ec49a326ada479400f4, a49da3317dc34fca8acceea4f877ebb4; no voters:
I20260926 03:42:48.984899 22287 raft_consensus.cc:2781] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:42:48.985041 22287 raft_consensus.cc:482] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:42:48.985133 22287 raft_consensus.cc:3037] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:48.988188 22287 raft_consensus.cc:504] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:48.988850 22287 leader_election.cc:288] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [CANDIDATE]: Term 1 election: Requested vote from peers a49da3317dc34fca8acceea4f877ebb4 (127.1.221.130:40135), d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb (127.1.221.129:42883)
I20260926 03:42:48.989405 21958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8505a20f56be4fbbbd236f428079ce98" candidate_uuid: "20e0089b04594ec49a326ada479400f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb"
I20260926 03:42:48.989614 21958 raft_consensus.cc:3037] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:48.991797 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8505a20f56be4fbbbd236f428079ce98" candidate_uuid: "20e0089b04594ec49a326ada479400f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49da3317dc34fca8acceea4f877ebb4"
I20260926 03:42:48.992046 22089 raft_consensus.cc:3037] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:48.992843 21958 raft_consensus.cc:2454] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e0089b04594ec49a326ada479400f4 in term 1.
I20260926 03:42:48.993379 22155 leader_election.cc:302] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e0089b04594ec49a326ada479400f4, d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb; no voters:
I20260926 03:42:48.993724 22287 raft_consensus.cc:2781] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:48.995271 22089 raft_consensus.cc:2454] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e0089b04594ec49a326ada479400f4 in term 1.
I20260926 03:42:48.995841 22287 raft_consensus.cc:686] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 1 LEADER]: Becoming Leader. State: Replica: 20e0089b04594ec49a326ada479400f4, State: Running, Role: LEADER
I20260926 03:42:48.996438 22287 consensus_queue.cc:237] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } }
I20260926 03:42:49.001642 21817 catalog_manager.cc:5495] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20e0089b04594ec49a326ada479400f4 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "20e0089b04594ec49a326ada479400f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e0089b04594ec49a326ada479400f4" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 } health_report { overall_health: UNKNOWN } } }
I20260926 03:42:49.069893 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:49.064017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.1.221.131:32945)
0926 03:42:49.064198 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:49.064202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:49.064221 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:42:49.064391 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:49.064395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:49.064486 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:49.064616 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:42:49.064997 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.065731 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.067249 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.067878 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.067981 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:49.068468 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:42:49.068990 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:42:49.069106 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:49.069417 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 03:42:49.069665 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:42:49.078243 22089 raft_consensus.cc:1261] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 1 FOLLOWER]: Refusing update from remote peer 20e0089b04594ec49a326ada479400f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:49.078238 21958 raft_consensus.cc:1261] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 1 FOLLOWER]: Refusing update from remote peer 20e0089b04594ec49a326ada479400f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:42:49.079254 22287 consensus_queue.cc:1035] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49da3317dc34fca8acceea4f877ebb4" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:49.079612 22292 consensus_queue.cc:1035] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 42883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:42:49.092954 22295 mvcc.cc:205] Tried to move back new op lower bound from 7333454516534153216 to 7333454516214067200. Current Snapshot: MvccSnapshot[applied={T|T < 7333454516534153216}]
I20260926 03:42:49.117832 21816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:49.118172 21816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:42:49.122589 21816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f878648edf74343b7a59559c1097044: Sending DeleteTablet for 3 replicas of tablet 8505a20f56be4fbbbd236f428079ce98
I20260926 03:42:49.123919 22199 tablet_service.cc:1514] Processing DeleteTablet for tablet 8505a20f56be4fbbbd236f428079ce98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:49 UTC) from {username='slave'} at 127.0.0.1:35962
I20260926 03:42:49.124701 22069 tablet_service.cc:1514] Processing DeleteTablet for tablet 8505a20f56be4fbbbd236f428079ce98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:49 UTC) from {username='slave'} at 127.0.0.1:39638
I20260926 03:42:49.126056 22305 tablet_replica.cc:330] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: stopping tablet replica
I20260926 03:42:49.126129 22304 tablet_replica.cc:330] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: stopping tablet replica
I20260926 03:42:49.126279 21938 tablet_service.cc:1514] Processing DeleteTablet for tablet 8505a20f56be4fbbbd236f428079ce98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:42:49 UTC) from {username='slave'} at 127.0.0.1:33820
I20260926 03:42:49.126675 22305 raft_consensus.cc:2229] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:49.127354 22305 raft_consensus.cc:2258] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:49.128371 22304 raft_consensus.cc:2229] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:42:49.129683 22304 raft_consensus.cc:2258] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:49.130102 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21876
I20260926 03:42:49.133162 22305 ts_tablet_manager.cc:1905] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:49.141431 22304 ts_tablet_manager.cc:1905] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:49.143087 22305 ts_tablet_manager.cc:1918] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:49.144141 22306 tablet_replica.cc:330] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: stopping tablet replica
I20260926 03:42:49.144866 22306 raft_consensus.cc:2229] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:42:49.145658 22306 raft_consensus.cc:2258] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:42:49.148958 22306 ts_tablet_manager.cc:1905] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:42:49.146131 22305 log.cc:1196] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-1/wal/wals/8505a20f56be4fbbbd236f428079ce98
I20260926 03:42:49.152333 22305 ts_tablet_manager.cc:1939] T 8505a20f56be4fbbbd236f428079ce98 P a49da3317dc34fca8acceea4f877ebb4: Deleting consensus metadata
I20260926 03:42:49.154767 21803 catalog_manager.cc:4841] TS a49da3317dc34fca8acceea4f877ebb4 (127.1.221.130:40135): tablet 8505a20f56be4fbbbd236f428079ce98 (table test-table [id=d7bf180a0dd84cf089e0e37255118b4d]) successfully deleted
I20260926 03:42:49.157668 22306 ts_tablet_manager.cc:1918] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:49.157914 22306 log.cc:1196] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-0/wal/wals/8505a20f56be4fbbbd236f428079ce98
I20260926 03:42:49.158245 22304 ts_tablet_manager.cc:1918] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:42:49.158527 22306 ts_tablet_manager.cc:1939] T 8505a20f56be4fbbbd236f428079ce98 P d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb: Deleting consensus metadata
I20260926 03:42:49.158555 22304 log.cc:1196] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Deleting WAL directory at /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393848592298-1910-0/minicluster-data/ts-2/wal/wals/8505a20f56be4fbbbd236f428079ce98
I20260926 03:42:49.159233 22304 ts_tablet_manager.cc:1939] T 8505a20f56be4fbbbd236f428079ce98 P 20e0089b04594ec49a326ada479400f4: Deleting consensus metadata
I20260926 03:42:49.161059 21804 catalog_manager.cc:4841] TS d7d9a2bb9c424c1e9ea9fe2ca4a3c2eb (127.1.221.129:42883): tablet 8505a20f56be4fbbbd236f428079ce98 (table test-table [id=d7bf180a0dd84cf089e0e37255118b4d]) successfully deleted
I20260926 03:42:49.161464 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:49.133059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.1.221.129:42883)
0926 03:42:49.133288 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:49.133295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:49.152242 (+ 18947us) server_negotiation.cc:407] Connection header received
0926 03:42:49.152602 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:49.152609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:49.152737 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:49.152824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:49.153574 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.154765 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.156823 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.157460 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.157564 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:49.159303 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:49.159316 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:49.159338 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:49.160431 (+ 1093us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:49.160512 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:49.160539 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:49.160646 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:49.160912 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 03:42:49.161069 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:42:49.161398 21801 catalog_manager.cc:4841] TS 20e0089b04594ec49a326ada479400f4 (127.1.221.131:32945): tablet 8505a20f56be4fbbbd236f428079ce98 (table test-table [id=d7bf180a0dd84cf089e0e37255118b4d]) successfully deleted
I20260926 03:42:49.162820 21998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:49.320891 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21876
I20260926 03:42:49.348280 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22007
I20260926 03:42:49.358804 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:49.349262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.1.221.130:40135)
0926 03:42:49.349536 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:49.349545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:49.349622 (+ 77us) server_negotiation.cc:407] Connection header received
0926 03:42:49.349719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:49.349726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:49.349878 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:49.350033 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:49.350668 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.351878 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.353564 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.354312 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.354483 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:49.356095 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:49.356153 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:49.356182 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:49.357513 (+ 1331us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:49.357617 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:49.357656 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:49.357865 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:49.358127 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 03:42:49.358306 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:42:49.359833 22129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:49.457669 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22007
I20260926 03:42:49.483994 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22137
I20260926 03:42:49.491761 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:49.484860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.1.221.131:32945)
0926 03:42:49.485112 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:49.485119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:49.485139 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:42:49.485213 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:49.485217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:49.485316 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:49.485406 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:49.485824 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.486774 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.488412 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:49.488889 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:49.489016 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:49.490157 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:49.490166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:49.490184 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:49.490890 (+ 706us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:49.490959 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:49.490982 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:49.491078 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:49.491242 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 03:42:49.491351 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:42:49.492892 22259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:49.626272 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22137
I20260926 03:42:49.645124 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 21786
I20260926 03:42:49.646389 21847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:49.767553 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 21786
2026-09-26T03:42:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3341 ms)
[----------] 6 tests from EncryptionPolicyTest (23737 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:49 dist-test-slave-lqrk krb5kdc[22314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:49 dist-test-slave-lqrk krb5kdc[22314](info): set up 2 sockets
Sep 26 03:42:49 dist-test-slave-lqrk krb5kdc[22314](info): commencing operation
krb5kdc: starting...
W20260926 03:42:50.872577 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:51 dist-test-slave-lqrk krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:42:51.187331 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:51.565291 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:51.565614 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:51.565740 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:51.573675 22327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:51.573819 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:51.573868 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:51.573961 22327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:51.574019 22327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:51.585330 22327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:51.586758 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:51.588821 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:51.603765 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:51.603834 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:51.603801 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:51.604647 22327 server_base.cc:1034] running on GCE node
I20260926 03:42:51.605530 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:51.607110 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:51.608347 22327 hybrid_clock.cc:648] HybridClock initialized: now 1790394171608283 us; error 54 us; skew 500 ppm
I20260926 03:42:51.608767 22327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:51 dist-test-slave-lqrk krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:51.612349 22327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:42:51.614076 22327 webserver.cc:460] Webserver started at http://127.0.0.1:43765/ using document root <none> and password file <none>
I20260926 03:42:51.614836 22327 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:51.614962 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:51.615288 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:51.618711 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "97762257978449b2bd5aa9b17cc51636"
format_stamp: "Formatted at 2026-09-26 03:42:51 on dist-test-slave-lqrk"
server_key: "18a0977c0b2e59d894af8dc6fba514f4"
server_key_iv: "0ec70b38cb32d795143f2cbcd4556ec6"
server_key_version: "encryptionkey@0"
I20260926 03:42:51.619659 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "97762257978449b2bd5aa9b17cc51636"
format_stamp: "Formatted at 2026-09-26 03:42:51 on dist-test-slave-lqrk"
server_key: "18a0977c0b2e59d894af8dc6fba514f4"
server_key_iv: "0ec70b38cb32d795143f2cbcd4556ec6"
server_key_version: "encryptionkey@0"
I20260926 03:42:51.625458 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:42:51.629390 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:51.631002 22327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:42:51.631186 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "97762257978449b2bd5aa9b17cc51636"
format_stamp: "Formatted at 2026-09-26 03:42:51 on dist-test-slave-lqrk"
server_key: "18a0977c0b2e59d894af8dc6fba514f4"
server_key_iv: "0ec70b38cb32d795143f2cbcd4556ec6"
server_key_version: "encryptionkey@0"
I20260926 03:42:51.631469 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:51.661202 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:51.666556 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:51.666885 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:51.692515 22327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35075
I20260926 03:42:51.692559 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35075 every 8 connection(s)
I20260926 03:42:51.694634 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:51.699100 22394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:51.702633 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22327
I20260926 03:42:51.703001 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:51.703694 1910 external_mini_cluster.cc:1428] Setting key 328abd56210473f2be85a7ecd18f3ede
I20260926 03:42:51.713512 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636: Bootstrap starting.
I20260926 03:42:51.717756 22394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:51.719249 22394 log.cc:824] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:42:51 dist-test-slave-lqrk krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:42:51.723703 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636: No bootstrap required, opened a new log
I20260926 03:42:51.735214 22394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } }
I20260926 03:42:51.735699 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:51.706109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45611 (local address 127.0.0.1:35075)
0926 03:42:51.707027 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:51.707039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:51.707082 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:42:51.708857 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:51.708884 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:51.709382 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:51.709762 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:51.711886 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:51.713402 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:51.715421 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:51.716301 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:51.716484 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:51.723035 (+ 6551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:51.723062 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:51.723092 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:51.723171 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:51.729135 (+ 5964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:51.730507 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:51.730516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:51.730525 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:51.730791 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:51.731938 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:51.731942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:51.731945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:51.732634 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:51.732942 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:51.733883 (+ 941us) server_negotiation.cc:299] Negotiation successful
0926 03:42:51.734347 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":459,"threads_started":1}
I20260926 03:42:51.735743 22394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:51.735895 22394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97762257978449b2bd5aa9b17cc51636, State: Initialized, Role: FOLLOWER
I20260926 03:42:51.736755 22394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } }
I20260926 03:42:51.737043 22394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:51.737213 22394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:51.737408 22394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:51.740314 22394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } }
I20260926 03:42:51.740541 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:42:51.741111 22394 leader_election.cc:302] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97762257978449b2bd5aa9b17cc51636; no voters:
I20260926 03:42:51.741832 22394 leader_election.cc:288] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:51.741937 22399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:51.743676 22399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [term 1 LEADER]: Becoming Leader. State: Replica: 97762257978449b2bd5aa9b17cc51636, State: Running, Role: LEADER
I20260926 03:42:51.744321 22394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:51.744293 22399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } }
I20260926 03:42:51.749445 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97762257978449b2bd5aa9b17cc51636. Latest consensus state: current_term: 1 leader_uuid: "97762257978449b2bd5aa9b17cc51636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } } }
I20260926 03:42:51.749943 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:51.749529 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97762257978449b2bd5aa9b17cc51636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97762257978449b2bd5aa9b17cc51636" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35075 } } }
I20260926 03:42:51.750110 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:51.766600 22408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:51.772382 22408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:51.786866 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:51.765662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52279 (local address 127.0.0.1:35075)
0926 03:42:51.765869 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:51.765876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:51.766846 (+ 970us) server_negotiation.cc:407] Connection header received
0926 03:42:51.767178 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:51.767184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:51.767281 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:51.767485 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:51.769677 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:51.771010 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:51.773103 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:51.773875 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:51.773997 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:51.777782 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:51.777790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:51.777796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:51.777851 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:51.781771 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:51.783007 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:51.783014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:51.783018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:51.783212 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:51.783879 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:51.783885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:51.783889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:51.784420 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:51.784747 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:51.786101 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0926 03:42:51.786418 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:42:51.792387 22408 catalog_manager.cc:1347] Generated new cluster ID: 8e81f4f71b614628ac364bb105ce68f6
I20260926 03:42:51.792532 22408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:51.802613 22408 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:51.804037 22408 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:51.810395 22408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97762257978449b2bd5aa9b17cc51636: Generated new TSK 0
I20260926 03:42:51.811218 22408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:51.823701 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22327
I20260926 03:42:51.825038 22389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:51.918963 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22327
W20260926 03:42:51.934316 22312 connection.cc:535] client connection to 127.0.0.1:35075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35075): Connection reset by peer (error 104)
2026-09-26T03:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:52 dist-test-slave-lqrk krb5kdc[22426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:52 dist-test-slave-lqrk krb5kdc[22426](info): set up 2 sockets
Sep 26 03:42:52 dist-test-slave-lqrk krb5kdc[22426](info): commencing operation
krb5kdc: starting...
W20260926 03:42:52.976058 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:53 dist-test-slave-lqrk krb5kdc[22426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:42:53.271773 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:53.619645 22439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:53.619976 22439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:53.620115 22439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:53.627661 22439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:53.627802 22439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:53.627882 22439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:53.627966 22439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:53.628031 22439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:53.638188 22439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:53.639623 22439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:53.640944 22439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:53.652638 22444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:53.652714 22445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:53.652769 22447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:53.653638 22439 server_base.cc:1034] running on GCE node
I20260926 03:42:53.654516 22439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:53.656323 22439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:53.657618 22439 hybrid_clock.cc:648] HybridClock initialized: now 1790394173657541 us; error 59 us; skew 500 ppm
I20260926 03:42:53.658038 22439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:53 dist-test-slave-lqrk krb5kdc[22426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:53.661401 22439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:42:53.662968 22439 webserver.cc:460] Webserver started at http://127.0.0.1:33213/ using document root <none> and password file <none>
I20260926 03:42:53.663733 22439 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:53.663847 22439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:53.664094 22439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:53.667132 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "3985d456d6cf4f559f2183b51577d432"
format_stamp: "Formatted at 2026-09-26 03:42:53 on dist-test-slave-lqrk"
server_key: "2c2a67a6ef8141669b57837102cedbff"
server_key_iv: "008ac13e9fc45910d6625010b6bddf64"
server_key_version: "encryptionkey@0"
I20260926 03:42:53.667930 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "3985d456d6cf4f559f2183b51577d432"
format_stamp: "Formatted at 2026-09-26 03:42:53 on dist-test-slave-lqrk"
server_key: "2c2a67a6ef8141669b57837102cedbff"
server_key_iv: "008ac13e9fc45910d6625010b6bddf64"
server_key_version: "encryptionkey@0"
I20260926 03:42:53.673341 22439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:42:53.677009 22454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:53.678438 22439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:42:53.678594 22439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "3985d456d6cf4f559f2183b51577d432"
format_stamp: "Formatted at 2026-09-26 03:42:53 on dist-test-slave-lqrk"
server_key: "2c2a67a6ef8141669b57837102cedbff"
server_key_iv: "008ac13e9fc45910d6625010b6bddf64"
server_key_version: "encryptionkey@0"
I20260926 03:42:53.678779 22439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:53.711858 22439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:53.716470 22439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:53.716718 22439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:53.737238 22439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42511
I20260926 03:42:53.737251 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42511 every 8 connection(s)
I20260926 03:42:53.738680 22439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:53.742691 22506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:53.748238 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22439
I20260926 03:42:53.748430 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:53.748955 1910 external_mini_cluster.cc:1428] Setting key 06004d8cc5ab6b4cb17da95b28e4f1d5
I20260926 03:42:53.755429 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432: Bootstrap starting.
I20260926 03:42:53.758904 22506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:53.760161 22506 log.cc:824] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:53.763947 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432: No bootstrap required, opened a new log
I20260926 03:42:53.769722 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:53.756057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43611 (local address 127.0.0.1:42511)
0926 03:42:53.756869 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:53.756897 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:53.756946 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:42:53.758949 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:53.758973 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:53.759502 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:53.759896 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:53.760697 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:53.762565 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:53.764606 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:53.765399 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:53.765549 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:53.766995 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:53.767004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:53.767023 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:53.767677 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:53.767774 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:53.767805 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:53.767951 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:53.768362 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 03:42:53.768569 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":376,"threads_started":1}
I20260926 03:42:53.773234 22506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260926 03:42:53.773658 22506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:53.773813 22506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3985d456d6cf4f559f2183b51577d432, State: Initialized, Role: FOLLOWER
I20260926 03:42:53.774062 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:42:53.774459 22506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260926 03:42:53.774716 22506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:53.774861 22506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:53.775038 22506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:53.777729 22506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260926 03:42:53.778368 22506 leader_election.cc:302] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3985d456d6cf4f559f2183b51577d432; no voters:
I20260926 03:42:53.779079 22506 leader_election.cc:288] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:53.779186 22516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:53.779980 22516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [term 1 LEADER]: Becoming Leader. State: Replica: 3985d456d6cf4f559f2183b51577d432, State: Running, Role: LEADER
I20260926 03:42:53.780771 22516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } }
I20260926 03:42:53.783883 22518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3985d456d6cf4f559f2183b51577d432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } } }
I20260926 03:42:53.784336 22518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:53.785943 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:53.778454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59275 (local address 127.0.0.1:42511)
0926 03:42:53.778668 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:53.778676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:53.779025 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:42:53.779271 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:53.779275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:53.779417 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:53.779519 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:53.780181 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:53.781059 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:53.782619 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:53.783298 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:53.783442 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:53.784971 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:53.784978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:53.784983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:53.785007 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:53.785084 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:53.785108 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:53.785191 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:53.785503 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:42:53.785572 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:42:53.786170 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3985d456d6cf4f559f2183b51577d432. Latest consensus state: current_term: 1 leader_uuid: "3985d456d6cf4f559f2183b51577d432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3985d456d6cf4f559f2183b51577d432" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42511 } } }
I20260926 03:42:53.787181 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:53.794044 22506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:53.794260 22520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:53.799474 22520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:53.813007 22520 catalog_manager.cc:1347] Generated new cluster ID: 89a19912c1724bd4b5f6600f50a04f57
I20260926 03:42:53.813143 22520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:53.823518 22520 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:53.825112 22520 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:53.839123 22520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3985d456d6cf4f559f2183b51577d432: Generated new TSK 0
I20260926 03:42:53.840132 22520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:53.853420 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22439
W20260926 03:42:53.854743 22501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43611
W20260926 03:42:53.855769 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:42:53.856236 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22439
W20260926 03:42:53.871654 22424 connection.cc:535] client connection to 127.0.0.1:42511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42511): Connection reset by peer (error 104)
2026-09-26T03:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:53 dist-test-slave-lqrk krb5kdc[22538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:53 dist-test-slave-lqrk krb5kdc[22538](info): set up 2 sockets
Sep 26 03:42:53 dist-test-slave-lqrk krb5kdc[22538](info): commencing operation
krb5kdc: starting...
W20260926 03:42:54.903906 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:55 dist-test-slave-lqrk krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:42:55.189273 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:55.565039 22551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:55.565397 22551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:55.565537 22551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:55.574903 22551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:55.575079 22551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:55.575153 22551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:55.575249 22551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:55.575311 22551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:55.586344 22551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:55.587841 22551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:55.589195 22551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:55.604260 22556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:55.604168 22557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:55.604141 22559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:55.605479 22551 server_base.cc:1034] running on GCE node
I20260926 03:42:55.606472 22551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:55.608378 22551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:55.609860 22551 hybrid_clock.cc:648] HybridClock initialized: now 1790394175609755 us; error 78 us; skew 500 ppm
I20260926 03:42:55.610436 22551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:55 dist-test-slave-lqrk krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:55.614985 22551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:42:55.617141 22551 webserver.cc:460] Webserver started at http://127.0.0.1:43891/ using document root <none> and password file <none>
I20260926 03:42:55.617985 22551 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:55.618141 22551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:55.618460 22551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:55.621812 22551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "6c50d05984ac4423857cf4fb5d7bb884"
format_stamp: "Formatted at 2026-09-26 03:42:55 on dist-test-slave-lqrk"
server_key: "732df35ce3f8a800ec2043fe04f20660"
server_key_iv: "1fcbb7ed0b2baa5bda96aeae0c1a2d66"
server_key_version: "encryptionkey@0"
I20260926 03:42:55.622668 22551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "6c50d05984ac4423857cf4fb5d7bb884"
format_stamp: "Formatted at 2026-09-26 03:42:55 on dist-test-slave-lqrk"
server_key: "732df35ce3f8a800ec2043fe04f20660"
server_key_iv: "1fcbb7ed0b2baa5bda96aeae0c1a2d66"
server_key_version: "encryptionkey@0"
I20260926 03:42:55.629660 22551 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 03:42:55.634656 22566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:55.637033 22551 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 03:42:55.637435 22551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "6c50d05984ac4423857cf4fb5d7bb884"
format_stamp: "Formatted at 2026-09-26 03:42:55 on dist-test-slave-lqrk"
server_key: "732df35ce3f8a800ec2043fe04f20660"
server_key_iv: "1fcbb7ed0b2baa5bda96aeae0c1a2d66"
server_key_version: "encryptionkey@0"
I20260926 03:42:55.637683 22551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:55.646142 22551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:55.651731 22551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:55.652091 22551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:55.674906 22551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34637
I20260926 03:42:55.675001 22617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34637 every 8 connection(s)
I20260926 03:42:55.676573 22551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:55.679600 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22551
I20260926 03:42:55.679829 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:55.680509 1910 external_mini_cluster.cc:1428] Setting key 5907d976c9d2822ac60a69d42ed82c4a
I20260926 03:42:55.683025 22618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:55.691056 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:55.683009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53797 (local address 127.0.0.1:34637)
0926 03:42:55.684119 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:55.684133 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:55.684189 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:42:55.686138 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:55.686178 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:55.686710 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:55.687361 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:55.688939 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:55.688946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:55.688963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:55.688982 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:55.689093 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:55.689132 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:55.689369 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:55.689491 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 03:42:55.689739 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":572,"threads_started":1}
I20260926 03:42:55.697484 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:42:55.699642 22618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884: Bootstrap starting.
I20260926 03:42:55.704458 22618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:55.705503 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:55.702510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41871 (local address 127.0.0.1:34637)
0926 03:42:55.702724 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:55.702730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:55.703295 (+ 565us) server_negotiation.cc:407] Connection header received
0926 03:42:55.703535 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:55.703539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:55.703595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:55.703690 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:55.704723 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:55.704729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:55.704734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:42:55.704752 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:55.704797 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:42:55.704815 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:55.704945 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:55.705027 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 03:42:55.705098 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:42:55.706103 22618 log.cc:824] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:55.710443 22618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884: No bootstrap required, opened a new log
I20260926 03:42:55.720211 22618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } }
I20260926 03:42:55.720924 22618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:55.721127 22618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c50d05984ac4423857cf4fb5d7bb884, State: Initialized, Role: FOLLOWER
I20260926 03:42:55.721946 22618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } }
I20260926 03:42:55.722216 22618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:55.722371 22618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:55.722553 22618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:55.725365 22618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } }
I20260926 03:42:55.726274 22618 leader_election.cc:302] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c50d05984ac4423857cf4fb5d7bb884; no voters:
I20260926 03:42:55.727005 22618 leader_election.cc:288] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:55.727203 22628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:55.728111 22628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [term 1 LEADER]: Becoming Leader. State: Replica: 6c50d05984ac4423857cf4fb5d7bb884, State: Running, Role: LEADER
I20260926 03:42:55.728796 22628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } }
I20260926 03:42:55.729696 22618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:55.731956 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c50d05984ac4423857cf4fb5d7bb884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } } }
I20260926 03:42:55.733160 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:55.732517 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c50d05984ac4423857cf4fb5d7bb884. Latest consensus state: current_term: 1 leader_uuid: "6c50d05984ac4423857cf4fb5d7bb884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c50d05984ac4423857cf4fb5d7bb884" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34637 } } }
I20260926 03:42:55.733892 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:55.736649 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:55.742048 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:55.754201 22637 catalog_manager.cc:1347] Generated new cluster ID: bdfd73d0b81f46a0bcf7fd87f000a57c
I20260926 03:42:55.754456 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:55.776885 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:55.777859 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:55.787644 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c50d05984ac4423857cf4fb5d7bb884: Generated new TSK 0
I20260926 03:42:55.788879 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:55.795948 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22551
W20260926 03:42:55.798036 22613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53797
W20260926 03:42:55.799181 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:42:55.799410 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22551
2026-09-26T03:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:55 dist-test-slave-lqrk krb5kdc[22649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:55 dist-test-slave-lqrk krb5kdc[22649](info): set up 2 sockets
Sep 26 03:42:55 dist-test-slave-lqrk krb5kdc[22649](info): commencing operation
krb5kdc: starting...
W20260926 03:42:56.884421 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:57 dist-test-slave-lqrk krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:42:57.212556 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:57.618731 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:57.619089 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:57.619230 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:42:57.628782 22662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:42:57.628966 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:42:57.629040 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:42:57.629153 22662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:42:57.629220 22662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:42:57.640501 22662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:42:57.641983 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:42:57.643483 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:42:57.659049 22667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:57.659052 22670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:42:57.659088 22668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:42:57.660378 22662 server_base.cc:1034] running on GCE node
I20260926 03:42:57.661548 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:42:57.664278 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:42:57.667836 22662 hybrid_clock.cc:648] HybridClock initialized: now 1790394177667674 us; error 103 us; skew 500 ppm
I20260926 03:42:57.668877 22662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:42:57 dist-test-slave-lqrk krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:42:57.679323 22662 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:42:57.681931 22662 webserver.cc:460] Webserver started at http://127.0.0.1:33323/ using document root <none> and password file <none>
I20260926 03:42:57.683106 22662 fs_manager.cc:359] Metadata directory not provided
I20260926 03:42:57.683285 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:42:57.683946 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:42:57.689625 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "53d81518a4594a329ca5e15e324c88e7"
format_stamp: "Formatted at 2026-09-26 03:42:57 on dist-test-slave-lqrk"
server_key: "91e540924f78d98373d22c80d8e8e9dc"
server_key_iv: "b049811b4510718cbf35341dcd108ff8"
server_key_version: "encryptionkey@0"
I20260926 03:42:57.691279 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "53d81518a4594a329ca5e15e324c88e7"
format_stamp: "Formatted at 2026-09-26 03:42:57 on dist-test-slave-lqrk"
server_key: "91e540924f78d98373d22c80d8e8e9dc"
server_key_iv: "b049811b4510718cbf35341dcd108ff8"
server_key_version: "encryptionkey@0"
I20260926 03:42:57.700781 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260926 03:42:57.706391 22677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:42:57.708844 22662 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260926 03:42:57.709116 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "53d81518a4594a329ca5e15e324c88e7"
format_stamp: "Formatted at 2026-09-26 03:42:57 on dist-test-slave-lqrk"
server_key: "91e540924f78d98373d22c80d8e8e9dc"
server_key_iv: "b049811b4510718cbf35341dcd108ff8"
server_key_version: "encryptionkey@0"
I20260926 03:42:57.709383 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:42:57.729851 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:42:57.734740 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:42:57.735041 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:42:57.762121 22662 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36531
I20260926 03:42:57.762221 22728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36531 every 8 connection(s)
I20260926 03:42:57.763887 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:42:57.772158 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22662
I20260926 03:42:57.772467 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:42:57.773118 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:42:57.773460 1910 external_mini_cluster.cc:1428] Setting key bbcf6ab86552f3a959f806aaf2c2c3f6
Sep 26 03:42:57 dist-test-slave-lqrk krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:42:57.792339 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7: Bootstrap starting.
I20260926 03:42:57.797219 22729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7: Neither blocks nor log segments found. Creating new log.
I20260926 03:42:57.798830 22729 log.cc:824] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:42:57.803339 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7: No bootstrap required, opened a new log
I20260926 03:42:57.804385 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:57.776592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57621 (local address 127.0.0.1:36531)
0926 03:42:57.777893 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:57.777915 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:57.777980 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:42:57.780675 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:57.780724 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:57.781342 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:57.781847 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:57.784194 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:57.785679 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:57.787212 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:57.788149 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:57.788420 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:57.794159 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:57.794170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:57.794191 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:57.794261 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:57.797949 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:57.799265 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:57.799275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:57.799284 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:57.799593 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:57.800160 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:57.800165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:57.800168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:57.801039 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:57.801289 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:57.802073 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 03:42:57.802564 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":607,"threads_started":1}
I20260926 03:42:57.810456 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:42:57.816129 22729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260926 03:42:57.817160 22729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:42:57.817318 22729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d81518a4594a329ca5e15e324c88e7, State: Initialized, Role: FOLLOWER
I20260926 03:42:57.818270 22729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260926 03:42:57.818835 22729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:42:57.819025 22729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:42:57.819276 22729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:42:57.822978 22729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260926 03:42:57.824003 22729 leader_election.cc:302] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d81518a4594a329ca5e15e324c88e7; no voters:
I20260926 03:42:57.825327 22739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:42:57.825577 22729 leader_election.cc:288] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:42:57.827698 22739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [term 1 LEADER]: Becoming Leader. State: Replica: 53d81518a4594a329ca5e15e324c88e7, State: Running, Role: LEADER
I20260926 03:42:57.828959 22729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:42:57.829286 22739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260926 03:42:57.833743 22740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d81518a4594a329ca5e15e324c88e7. Latest consensus state: current_term: 1 leader_uuid: "53d81518a4594a329ca5e15e324c88e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260926 03:42:57.833689 22741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d81518a4594a329ca5e15e324c88e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d81518a4594a329ca5e15e324c88e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260926 03:42:57.834441 22740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:57.834498 22741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:42:57.841001 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:42:57.818363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36653 (local address 127.0.0.1:36531)
0926 03:42:57.818612 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:42:57.818618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:42:57.819203 (+ 585us) server_negotiation.cc:407] Connection header received
0926 03:42:57.819514 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:42:57.819520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:42:57.819616 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:42:57.819896 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:42:57.822107 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:57.822911 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:57.825254 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:42:57.826186 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:42:57.826461 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:42:57.831098 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:42:57.831107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:42:57.831111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:42:57.831185 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:42:57.836242 (+ 5057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:57.837752 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:57.837760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:57.837764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:57.838006 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:42:57.838561 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:42:57.838567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:42:57.838571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:42:57.839088 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:42:57.839464 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 03:42:57.840207 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 03:42:57.840527 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:42:57.845053 22747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:42:57.852599 22747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:42:57.866645 22747 catalog_manager.cc:1347] Generated new cluster ID: 3b02d0e5c8a7445995945bd57e0a4b8b
I20260926 03:42:57.866844 22747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:42:57.884312 22747 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:42:57.886673 22747 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:42:57.904191 22747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53d81518a4594a329ca5e15e324c88e7: Generated new TSK 0
I20260926 03:42:57.906908 22747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:42:57.926620 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22662
I20260926 03:42:57.928680 22724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:42:58.036316 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22662
2026-09-26T03:42:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:42:58 dist-test-slave-lqrk krb5kdc[22761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:42:58 dist-test-slave-lqrk krb5kdc[22761](info): set up 2 sockets
Sep 26 03:42:58 dist-test-slave-lqrk krb5kdc[22761](info): commencing operation
krb5kdc: starting...
W20260926 03:42:59.171936 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:42:59 dist-test-slave-lqrk krb5kdc[22761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:42:59.501293 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:42:59.992379 22774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:42:59.992758 22774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:42:59.992903 22774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:00.001977 22774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:00.002154 22774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:00.002204 22774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:00.002274 22774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:00.002360 22774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:00.017712 22774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:00.019729 22774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:00.021559 22774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:00.040884 22779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:00.041417 22782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:00.041010 22780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:00.044157 22774 server_base.cc:1034] running on GCE node
I20260926 03:43:00.045662 22774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:00.048874 22774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:00.052579 22774 hybrid_clock.cc:648] HybridClock initialized: now 1790394180052504 us; error 200 us; skew 500 ppm
I20260926 03:43:00.053457 22774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:00 dist-test-slave-lqrk krb5kdc[22761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:00.058637 22774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:00.061686 22774 webserver.cc:460] Webserver started at http://127.0.0.1:43021/ using document root <none> and password file <none>
I20260926 03:43:00.062989 22774 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:00.063211 22774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:00.063858 22774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:00.068486 22774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "e756a0a1d8154d91839e92028a06aca3"
format_stamp: "Formatted at 2026-09-26 03:43:00 on dist-test-slave-lqrk"
server_key: "ed38a40758f38807d850ff38c1071024"
server_key_iv: "88c0ca229c5b4a12efc9311b042a0d67"
server_key_version: "encryptionkey@0"
I20260926 03:43:00.069631 22774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "e756a0a1d8154d91839e92028a06aca3"
format_stamp: "Formatted at 2026-09-26 03:43:00 on dist-test-slave-lqrk"
server_key: "ed38a40758f38807d850ff38c1071024"
server_key_iv: "88c0ca229c5b4a12efc9311b042a0d67"
server_key_version: "encryptionkey@0"
I20260926 03:43:00.077646 22774 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 03:43:00.083499 22789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:00.085877 22774 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:43:00.086190 22774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "e756a0a1d8154d91839e92028a06aca3"
format_stamp: "Formatted at 2026-09-26 03:43:00 on dist-test-slave-lqrk"
server_key: "ed38a40758f38807d850ff38c1071024"
server_key_iv: "88c0ca229c5b4a12efc9311b042a0d67"
server_key_version: "encryptionkey@0"
I20260926 03:43:00.086549 22774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:00.104503 22774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:00.113619 22774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:00.114081 22774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:00.142835 22774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41171
I20260926 03:43:00.142884 22840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41171 every 8 connection(s)
I20260926 03:43:00.145179 22774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:00.150115 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:00.153836 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22774
I20260926 03:43:00.154067 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:00.154549 1910 external_mini_cluster.cc:1428] Setting key c7128e2d72d9a22df27ad512eb2d3a0e
I20260926 03:43:00.165454 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3: Bootstrap starting.
I20260926 03:43:00.170514 22841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:00.172187 22841 log.cc:824] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:00.173719 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:00.157100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58811 (local address 127.0.0.1:41171)
0926 03:43:00.158156 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:00.158173 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:00.158232 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:43:00.160635 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:00.160674 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:00.161384 (+ 710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:00.161913 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:00.162773 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:00.164611 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:00.166968 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:00.168180 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:00.168466 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:00.170109 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:00.170120 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:00.170147 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:00.171045 (+ 898us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:00.171183 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:00.171221 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:00.171366 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:00.171791 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 03:43:00.172236 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":467,"threads_started":1}
I20260926 03:43:00.175922 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3: No bootstrap required, opened a new log
I20260926 03:43:00.179231 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:00.184721 22841 raft_consensus.cc:348] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260926 03:43:00.185320 22841 raft_consensus.cc:374] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:00.185494 22841 raft_consensus.cc:729] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e756a0a1d8154d91839e92028a06aca3, State: Initialized, Role: FOLLOWER
I20260926 03:43:00.186314 22841 consensus_queue.cc:260] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260926 03:43:00.186594 22841 raft_consensus.cc:388] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:00.186751 22841 raft_consensus.cc:482] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:00.186990 22841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:00.190933 22841 raft_consensus.cc:504] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260926 03:43:00.191777 22841 leader_election.cc:302] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e756a0a1d8154d91839e92028a06aca3; no voters:
I20260926 03:43:00.192693 22841 leader_election.cc:288] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:00.192859 22851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:00.194103 22851 raft_consensus.cc:686] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [term 1 LEADER]: Becoming Leader. State: Replica: e756a0a1d8154d91839e92028a06aca3, State: Running, Role: LEADER
I20260926 03:43:00.195021 22851 consensus_queue.cc:237] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260926 03:43:00.195739 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:00.185023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41635 (local address 127.0.0.1:41171)
0926 03:43:00.185300 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:00.185312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:00.186020 (+ 708us) server_negotiation.cc:407] Connection header received
0926 03:43:00.186284 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:00.186289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:00.186385 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:00.186514 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:00.187304 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:00.188274 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:00.191061 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:00.192050 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:00.192223 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:00.194240 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:00.194255 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:00.194261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:00.194298 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:00.194383 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:00.194414 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:00.194500 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:00.194877 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:43:00.195041 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 03:43:00.198081 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e756a0a1d8154d91839e92028a06aca3. Latest consensus state: current_term: 1 leader_uuid: "e756a0a1d8154d91839e92028a06aca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260926 03:43:00.198576 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:00.199963 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e756a0a1d8154d91839e92028a06aca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e756a0a1d8154d91839e92028a06aca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260926 03:43:00.200347 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:00.202878 22841 sys_catalog.cc:564] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:00.203153 22855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:00.211107 22855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:00.224143 22855 catalog_manager.cc:1347] Generated new cluster ID: 5d8a9e58e801485896bf40ca1589fb56
I20260926 03:43:00.224296 22855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:00.252205 22855 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:00.254436 22855 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:00.264449 22855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e756a0a1d8154d91839e92028a06aca3: Generated new TSK 0
I20260926 03:43:00.265950 22855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:00.280820 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22774
W20260926 03:43:00.282419 22836 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58811
W20260926 03:43:00.283483 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22774: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:00.283707 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22774
2026-09-26T03:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:00 dist-test-slave-lqrk krb5kdc[22872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:00 dist-test-slave-lqrk krb5kdc[22872](info): set up 2 sockets
Sep 26 03:43:00 dist-test-slave-lqrk krb5kdc[22872](info): commencing operation
krb5kdc: starting...
W20260926 03:43:01.372466 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:01 dist-test-slave-lqrk krb5kdc[22872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:01.700766 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:02.094393 22885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:02.094755 22885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:02.094859 22885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:02.102988 22885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:02.103130 22885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:02.103204 22885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:02.103262 22885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:02.103315 22885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:02.116564 22885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:02.118562 22885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:02.120509 22885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:02.135215 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:02.135262 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:02.135179 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:02.136411 22885 server_base.cc:1034] running on GCE node
I20260926 03:43:02.137534 22885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:02.139424 22885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:02.140740 22885 hybrid_clock.cc:648] HybridClock initialized: now 1790394182140653 us; error 72 us; skew 500 ppm
I20260926 03:43:02.141387 22885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:02 dist-test-slave-lqrk krb5kdc[22872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:02.146190 22885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:02.148940 22885 webserver.cc:460] Webserver started at http://127.0.0.1:45451/ using document root <none> and password file <none>
I20260926 03:43:02.150383 22885 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:02.150599 22885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:02.150980 22885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:02.154487 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3"
format_stamp: "Formatted at 2026-09-26 03:43:02 on dist-test-slave-lqrk"
server_key: "68eb208aab05611d61a3fee126aa6677"
server_key_iv: "a40517e201ed0cf6f25cf2787c183d64"
server_key_version: "encryptionkey@0"
I20260926 03:43:02.155371 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3"
format_stamp: "Formatted at 2026-09-26 03:43:02 on dist-test-slave-lqrk"
server_key: "68eb208aab05611d61a3fee126aa6677"
server_key_iv: "a40517e201ed0cf6f25cf2787c183d64"
server_key_version: "encryptionkey@0"
I20260926 03:43:02.162501 22885 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260926 03:43:02.167245 22900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:02.169703 22885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:43:02.169978 22885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3"
format_stamp: "Formatted at 2026-09-26 03:43:02 on dist-test-slave-lqrk"
server_key: "68eb208aab05611d61a3fee126aa6677"
server_key_iv: "a40517e201ed0cf6f25cf2787c183d64"
server_key_version: "encryptionkey@0"
I20260926 03:43:02.170236 22885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:02.177376 22885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:02.182592 22885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:02.182926 22885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:02.204831 22885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40875
I20260926 03:43:02.204877 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40875 every 8 connection(s)
I20260926 03:43:02.206382 22885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:02.210328 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22885
I20260926 03:43:02.210500 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:02.210987 1910 external_mini_cluster.cc:1428] Setting key 42c10aa0812f4b374b89d4cb0c804c5d
I20260926 03:43:02.211927 22952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:02.221537 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:02.213605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49021 (local address 127.0.0.1:40875)
0926 03:43:02.214607 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:02.214621 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:02.214671 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:43:02.216435 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:02.216466 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:02.216952 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:02.217388 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:02.219029 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:02.219037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:02.219055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:02.219076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:02.219281 (+ 205us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:02.219323 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:02.219652 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:02.219798 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:43:02.220059 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":452,"threads_started":1}
I20260926 03:43:02.226413 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:02.229851 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: Bootstrap starting.
I20260926 03:43:02.234182 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:02.231343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55801 (local address 127.0.0.1:40875)
0926 03:43:02.231743 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:02.231750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:02.231894 (+ 144us) server_negotiation.cc:407] Connection header received
0926 03:43:02.232086 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:02.232093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:02.232197 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:02.232358 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:02.233272 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:02.233277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:02.233282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:02.233303 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:02.233363 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:02.233386 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:02.233545 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:02.233758 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:43:02.233836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260926 03:43:02.234926 22952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:02.236730 22952 log.cc:824] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:02.241036 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: No bootstrap required, opened a new log
I20260926 03:43:02.250180 22952 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260926 03:43:02.251282 22952 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:02.251492 22952 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4caa5b9f3bf408fb58b0739a07c2ab3, State: Initialized, Role: FOLLOWER
I20260926 03:43:02.252264 22952 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260926 03:43:02.252545 22952 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:02.252682 22952 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:02.252851 22952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:02.255370 22952 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260926 03:43:02.256099 22952 leader_election.cc:302] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4caa5b9f3bf408fb58b0739a07c2ab3; no voters:
I20260926 03:43:02.256871 22952 leader_election.cc:288] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:02.257138 22962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:02.258375 22962 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [term 1 LEADER]: Becoming Leader. State: Replica: a4caa5b9f3bf408fb58b0739a07c2ab3, State: Running, Role: LEADER
I20260926 03:43:02.259506 22952 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:02.259362 22962 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260926 03:43:02.266178 22964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4caa5b9f3bf408fb58b0739a07c2ab3. Latest consensus state: current_term: 1 leader_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260926 03:43:02.267859 22964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:02.266180 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4caa5b9f3bf408fb58b0739a07c2ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260926 03:43:02.268219 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3 [sys.catalog]: This master's current role is: LEADER
W20260926 03:43:02.279081 22978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:02.279287 22978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:02.279632 22973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:02.282205 22973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:02.296161 22973 catalog_manager.cc:1347] Generated new cluster ID: 32e82db54d254a0c87e27ef1ae13eed6
I20260926 03:43:02.296346 22973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:02.317107 22973 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:02.317819 22973 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:02.335484 22973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4caa5b9f3bf408fb58b0739a07c2ab3: Generated new TSK 0
I20260926 03:43:02.336897 22973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:02.356052 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22885
W20260926 03:43:02.359143 22947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49021
W20260926 03:43:02.360756 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:02.361025 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22885
2026-09-26T03:43:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:02 dist-test-slave-lqrk krb5kdc[22983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:02 dist-test-slave-lqrk krb5kdc[22983](info): set up 2 sockets
Sep 26 03:43:02 dist-test-slave-lqrk krb5kdc[22983](info): commencing operation
krb5kdc: starting...
W20260926 03:43:03.456530 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:03 dist-test-slave-lqrk krb5kdc[22983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:03.769477 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:04.195439 22996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:04.195883 22996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:04.195995 22996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:04.204556 22996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:04.204735 22996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:04.204797 22996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:04.204864 22996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:04.204916 22996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:04.216748 22996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:04.218593 22996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:04.220309 22996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:04.236621 23001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:04.236663 23004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:04.236593 23002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:04.237648 22996 server_base.cc:1034] running on GCE node
I20260926 03:43:04.238605 22996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:04.240450 22996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:04.241808 22996 hybrid_clock.cc:648] HybridClock initialized: now 1790394184241727 us; error 78 us; skew 500 ppm
I20260926 03:43:04.242273 22996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:04 dist-test-slave-lqrk krb5kdc[22983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:04.250741 22996 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:04.252804 22996 webserver.cc:460] Webserver started at http://127.0.0.1:34123/ using document root <none> and password file <none>
I20260926 03:43:04.253700 22996 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:04.253834 22996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:04.254163 22996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:04.257650 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "7ab961076d874d6c803f750c6e950499"
format_stamp: "Formatted at 2026-09-26 03:43:04 on dist-test-slave-lqrk"
server_key: "cee1c39e56c38f573ee3e4bb0a9e7172"
server_key_iv: "0c87689620adcd8e654faeb2a6929e14"
server_key_version: "encryptionkey@0"
I20260926 03:43:04.258944 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "7ab961076d874d6c803f750c6e950499"
format_stamp: "Formatted at 2026-09-26 03:43:04 on dist-test-slave-lqrk"
server_key: "cee1c39e56c38f573ee3e4bb0a9e7172"
server_key_iv: "0c87689620adcd8e654faeb2a6929e14"
server_key_version: "encryptionkey@0"
I20260926 03:43:04.266453 22996 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:43:04.271150 23011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:04.273134 22996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:43:04.273381 22996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "7ab961076d874d6c803f750c6e950499"
format_stamp: "Formatted at 2026-09-26 03:43:04 on dist-test-slave-lqrk"
server_key: "cee1c39e56c38f573ee3e4bb0a9e7172"
server_key_iv: "0c87689620adcd8e654faeb2a6929e14"
server_key_version: "encryptionkey@0"
I20260926 03:43:04.273607 22996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:04.293179 22996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:04.297977 22996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:04.298307 22996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:04.319353 22996 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35541
I20260926 03:43:04.319531 23062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35541 every 8 connection(s)
I20260926 03:43:04.320969 22996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:04.322685 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 22996
I20260926 03:43:04.322917 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:04.323586 1910 external_mini_cluster.cc:1428] Setting key e4cbe9b47ce9a57d14c9ce9120b45b58
I20260926 03:43:04.327204 23063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:43:04 dist-test-slave-lqrk krb5kdc[22983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:43:04.341264 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499: Bootstrap starting.
I20260926 03:43:04.345573 23063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:04.346949 23063 log.cc:824] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:04.349087 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:04.326039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:44437 (local address 127.0.0.1:35541)
0926 03:43:04.327119 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:04.327136 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:04.327195 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:43:04.329035 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:04.329063 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:04.329576 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:04.329991 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:04.331515 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:04.333016 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:04.334450 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:04.335243 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:04.335494 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:04.340379 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:04.340394 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:04.340415 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:04.340463 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:04.344103 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:04.345007 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:04.345015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:04.345023 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:04.345214 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:04.345669 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:04.345675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:04.345678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:04.346350 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:04.346728 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:04.347175 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 03:43:04.347599 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":416,"threads_started":1}
I20260926 03:43:04.351279 23063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499: No bootstrap required, opened a new log
I20260926 03:43:04.354812 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:04.361045 23063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260926 03:43:04.361562 23063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:04.361658 23063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab961076d874d6c803f750c6e950499, State: Initialized, Role: FOLLOWER
I20260926 03:43:04.362427 23063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260926 03:43:04.362728 23063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:04.362903 23063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:04.363116 23063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:04.366643 23063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260926 03:43:04.367547 23063 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab961076d874d6c803f750c6e950499; no voters:
I20260926 03:43:04.368407 23063 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:04.368654 23073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:04.371148 23063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:04.376049 23073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [term 1 LEADER]: Becoming Leader. State: Replica: 7ab961076d874d6c803f750c6e950499, State: Running, Role: LEADER
I20260926 03:43:04.376914 23073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260926 03:43:04.378283 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:04.359926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:45505 (local address 127.0.0.1:35541)
0926 03:43:04.360154 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:04.360160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:04.360547 (+ 387us) server_negotiation.cc:407] Connection header received
0926 03:43:04.360746 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:04.360751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:04.360820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:04.360909 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:04.362442 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:04.363450 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:04.365041 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:04.365781 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:04.365943 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:04.369447 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:04.369456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:04.369460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:04.369518 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:04.373820 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:04.375025 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:04.375032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:04.375036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:04.375230 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:04.375845 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:04.375850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:04.375854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:04.376263 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:04.376579 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:04.377691 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0926 03:43:04.377885 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:43:04.380752 23074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab961076d874d6c803f750c6e950499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260926 03:43:04.381300 23074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:04.382382 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab961076d874d6c803f750c6e950499. Latest consensus state: current_term: 1 leader_uuid: "7ab961076d874d6c803f750c6e950499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab961076d874d6c803f750c6e950499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260926 03:43:04.382975 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:04.385270 23080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:04.392350 23080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:04.404973 23080 catalog_manager.cc:1347] Generated new cluster ID: f14a4cfb87fc4485891fee3513e16fc3
I20260926 03:43:04.405117 23080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:04.418526 23080 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:04.420471 23080 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:04.434559 23080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab961076d874d6c803f750c6e950499: Generated new TSK 0
I20260926 03:43:04.435580 23080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:04.447050 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 22996
I20260926 03:43:04.448398 23058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:04.546372 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 22996
W20260926 03:43:04.561817 22981 connection.cc:535] client connection to 127.0.0.1:35541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35541): Connection reset by peer (error 104)
2026-09-26T03:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:04 dist-test-slave-lqrk krb5kdc[23095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:04 dist-test-slave-lqrk krb5kdc[23095](info): set up 2 sockets
Sep 26 03:43:04 dist-test-slave-lqrk krb5kdc[23095](info): commencing operation
krb5kdc: starting...
W20260926 03:43:05.599905 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:05 dist-test-slave-lqrk krb5kdc[23095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:05.912093 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:06.321132 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:06.321569 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:06.321686 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:06.330544 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:06.330710 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:06.330763 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:06.330828 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:06.330876 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:06.342691 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:06.344545 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:06.346240 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:06.361258 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:06.361311 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:06.361490 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:06.362978 23108 server_base.cc:1034] running on GCE node
I20260926 03:43:06.364151 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:06.366223 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:06.367578 23108 hybrid_clock.cc:648] HybridClock initialized: now 1790394186367457 us; error 60 us; skew 500 ppm
I20260926 03:43:06.368065 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:06 dist-test-slave-lqrk krb5kdc[23095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:06.372745 23108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:06.376541 23108 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
I20260926 03:43:06.377668 23108 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:06.377807 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:06.378160 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:06.382300 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "829c484ce63a4e3da216ade568afc664"
format_stamp: "Formatted at 2026-09-26 03:43:06 on dist-test-slave-lqrk"
server_key: "ccf0a8b18b3fef1ce440a359ad1cf38a"
server_key_iv: "9f53c9c663db231b1d5c94f97ba6bbe7"
server_key_version: "encryptionkey@0"
I20260926 03:43:06.383342 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "829c484ce63a4e3da216ade568afc664"
format_stamp: "Formatted at 2026-09-26 03:43:06 on dist-test-slave-lqrk"
server_key: "ccf0a8b18b3fef1ce440a359ad1cf38a"
server_key_iv: "9f53c9c663db231b1d5c94f97ba6bbe7"
server_key_version: "encryptionkey@0"
I20260926 03:43:06.390472 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:43:06.395826 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:06.398188 23108 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:43:06.398483 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "829c484ce63a4e3da216ade568afc664"
format_stamp: "Formatted at 2026-09-26 03:43:06 on dist-test-slave-lqrk"
server_key: "ccf0a8b18b3fef1ce440a359ad1cf38a"
server_key_iv: "9f53c9c663db231b1d5c94f97ba6bbe7"
server_key_version: "encryptionkey@0"
I20260926 03:43:06.398763 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:06.437098 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:06.442649 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:06.442978 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:06.465095 23108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37857
I20260926 03:43:06.465101 23174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37857 every 8 connection(s)
I20260926 03:43:06.466912 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:06.471446 23175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:06.475507 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23108
I20260926 03:43:06.475718 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:06.476217 1910 external_mini_cluster.cc:1428] Setting key e6da829ba115c536ce6a89738736d9a0
I20260926 03:43:06.484863 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: Bootstrap starting.
I20260926 03:43:06.488848 23175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:06.490227 23175 log.cc:824] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:06.494303 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: No bootstrap required, opened a new log
I20260926 03:43:06.494382 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:06.478868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:46965 (local address 127.0.0.1:37857)
0926 03:43:06.479890 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:06.479908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:06.479971 (+ 63us) server_negotiation.cc:407] Connection header received
0926 03:43:06.482874 (+ 2903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:06.482909 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:06.483612 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:06.484103 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:06.484851 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:06.486897 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:06.488696 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:06.489739 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:06.489945 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:06.491542 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:06.491551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:06.491567 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:06.492270 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:06.492375 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:06.492407 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:06.492584 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:06.492874 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 03:43:06.493055 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":454,"threads_started":1}
I20260926 03:43:06.498523 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:06.502539 23175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260926 03:43:06.503211 23175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:06.503345 23175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829c484ce63a4e3da216ade568afc664, State: Initialized, Role: FOLLOWER
I20260926 03:43:06.504128 23175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260926 03:43:06.504426 23175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:06.504590 23175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:06.504771 23175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:06.507807 23175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260926 03:43:06.508486 23175 leader_election.cc:302] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 829c484ce63a4e3da216ade568afc664; no voters:
I20260926 03:43:06.509325 23175 leader_election.cc:288] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:06.509415 23185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:06.510182 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:06.502785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:37829 (local address 127.0.0.1:37857)
0926 03:43:06.503001 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:06.503008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:06.503418 (+ 410us) server_negotiation.cc:407] Connection header received
0926 03:43:06.503659 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:06.503663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:06.503745 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:06.503878 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:06.504496 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:06.505389 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:06.507205 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:06.507713 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:06.507823 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:06.509200 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:06.509206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:06.509211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:06.509240 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:06.509296 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:06.509327 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:06.509465 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:06.509715 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 03:43:06.509786 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:43:06.511787 23175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:06.516351 23185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [term 1 LEADER]: Becoming Leader. State: Replica: 829c484ce63a4e3da216ade568afc664, State: Running, Role: LEADER
I20260926 03:43:06.516863 23185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260926 03:43:06.519618 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "829c484ce63a4e3da216ade568afc664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260926 03:43:06.519834 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 829c484ce63a4e3da216ade568afc664. Latest consensus state: current_term: 1 leader_uuid: "829c484ce63a4e3da216ade568afc664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829c484ce63a4e3da216ade568afc664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260926 03:43:06.520018 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:06.520061 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664 [sys.catalog]: This master's current role is: LEADER
W20260926 03:43:06.525482 23199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:06.525600 23199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:06.525753 23200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:06.527712 23200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:06.537860 23200 catalog_manager.cc:1347] Generated new cluster ID: f574c0a986af4e9fb7dfa448100726ed
I20260926 03:43:06.538002 23200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:06.553562 23200 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:06.555536 23200 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:06.563436 23200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 829c484ce63a4e3da216ade568afc664: Generated new TSK 0
I20260926 03:43:06.564677 23200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:06.571132 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23108
W20260926 03:43:06.572502 23170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:46965
W20260926 03:43:06.573710 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:06.573849 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23108
W20260926 03:43:06.589474 23093 connection.cc:535] client connection to 127.0.0.1:37857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37857): Connection reset by peer (error 104)
2026-09-26T03:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:06 dist-test-slave-lqrk krb5kdc[23206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:06 dist-test-slave-lqrk krb5kdc[23206](info): set up 2 sockets
Sep 26 03:43:06 dist-test-slave-lqrk krb5kdc[23206](info): commencing operation
krb5kdc: starting...
W20260926 03:43:07.616016 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:07 dist-test-slave-lqrk krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:07.889865 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:08.253755 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:08.254092 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:08.254223 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:08.262223 23219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:08.262383 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:08.262459 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:08.262538 23219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:08.262619 23219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:08.273329 23219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:08.274758 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:08.276494 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:08.289536 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:08.289461 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:08.289510 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:08.290795 23219 server_base.cc:1034] running on GCE node
I20260926 03:43:08.292093 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:08.293715 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:08.294963 23219 hybrid_clock.cc:648] HybridClock initialized: now 1790394188294901 us; error 48 us; skew 500 ppm
I20260926 03:43:08.295365 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:08 dist-test-slave-lqrk krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:08.298974 23219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:08.300976 23219 webserver.cc:460] Webserver started at http://127.0.0.1:36143/ using document root <none> and password file <none>
I20260926 03:43:08.301898 23219 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:08.302042 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:08.302373 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:08.305634 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "8d64d5f5d51d493d9726dd767a53933d"
format_stamp: "Formatted at 2026-09-26 03:43:08 on dist-test-slave-lqrk"
server_key: "df523de62f4180c9bdbad2e1a4894a2c"
server_key_iv: "860c347a7e846f0a9d4c6f7abf7ff5db"
server_key_version: "encryptionkey@0"
I20260926 03:43:08.306473 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "8d64d5f5d51d493d9726dd767a53933d"
format_stamp: "Formatted at 2026-09-26 03:43:08 on dist-test-slave-lqrk"
server_key: "df523de62f4180c9bdbad2e1a4894a2c"
server_key_iv: "860c347a7e846f0a9d4c6f7abf7ff5db"
server_key_version: "encryptionkey@0"
I20260926 03:43:08.312162 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 03:43:08.315980 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:08.317355 23219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:43:08.317523 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "8d64d5f5d51d493d9726dd767a53933d"
format_stamp: "Formatted at 2026-09-26 03:43:08 on dist-test-slave-lqrk"
server_key: "df523de62f4180c9bdbad2e1a4894a2c"
server_key_iv: "860c347a7e846f0a9d4c6f7abf7ff5db"
server_key_version: "encryptionkey@0"
I20260926 03:43:08.317764 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:08.324466 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:08.329339 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:08.329645 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:08.349506 23219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40365
I20260926 03:43:08.349545 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40365 every 8 connection(s)
I20260926 03:43:08.350970 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:08.354854 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:08.355950 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23219
I20260926 03:43:08.356130 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:08.356700 1910 external_mini_cluster.cc:1428] Setting key f57817cc056baae39790f8cb8ea36006
I20260926 03:43:08.365760 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:08.358833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:33591 (local address 127.0.0.1:40365)
0926 03:43:08.359848 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:08.359864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:08.360185 (+ 321us) server_negotiation.cc:407] Connection header received
0926 03:43:08.361881 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:08.361916 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:08.362418 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:08.362889 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:08.363884 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:08.363893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:08.363910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:08.363928 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:08.363999 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:08.364026 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:08.364246 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:08.364363 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 03:43:08.364555 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":434,"threads_started":1}
I20260926 03:43:08.369323 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d: Bootstrap starting.
I20260926 03:43:08.369407 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:08.373250 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:08.374555 23286 log.cc:824] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:08.377475 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:08.374675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:60075 (local address 127.0.0.1:40365)
0926 03:43:08.374852 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:08.374858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:08.375334 (+ 476us) server_negotiation.cc:407] Connection header received
0926 03:43:08.375537 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:08.375543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:08.375606 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:08.375693 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:08.376565 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:08.376572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:08.376576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:08.376593 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:08.376632 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:08.376647 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:08.377026 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:08.377136 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 03:43:08.377206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:43:08.377871 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d: No bootstrap required, opened a new log
I20260926 03:43:08.391124 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260926 03:43:08.391693 23286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:08.391805 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d64d5f5d51d493d9726dd767a53933d, State: Initialized, Role: FOLLOWER
I20260926 03:43:08.392541 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260926 03:43:08.392841 23286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:08.392987 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:08.393146 23286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:08.395484 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260926 03:43:08.396023 23286 leader_election.cc:302] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d64d5f5d51d493d9726dd767a53933d; no voters:
I20260926 03:43:08.396596 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:08.396762 23296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:08.397845 23296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [term 1 LEADER]: Becoming Leader. State: Replica: 8d64d5f5d51d493d9726dd767a53933d, State: Running, Role: LEADER
I20260926 03:43:08.398514 23296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260926 03:43:08.398794 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:08.401365 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d64d5f5d51d493d9726dd767a53933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260926 03:43:08.401551 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d64d5f5d51d493d9726dd767a53933d. Latest consensus state: current_term: 1 leader_uuid: "8d64d5f5d51d493d9726dd767a53933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64d5f5d51d493d9726dd767a53933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260926 03:43:08.402079 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:08.402077 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:08.405478 23304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:08.410319 23304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:08.420158 23304 catalog_manager.cc:1347] Generated new cluster ID: 50e418efe4c646ea8ee680c6b568d4e3
I20260926 03:43:08.420292 23304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:08.435420 23304 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:08.435995 23304 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:08.447010 23304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d64d5f5d51d493d9726dd767a53933d: Generated new TSK 0
I20260926 03:43:08.448246 23304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:08.464813 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23219
W20260926 03:43:08.467118 23281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:33591
W20260926 03:43:08.467976 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:08.468119 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23219
2026-09-26T03:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:08 dist-test-slave-lqrk krb5kdc[23317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:08 dist-test-slave-lqrk krb5kdc[23317](info): set up 2 sockets
Sep 26 03:43:08 dist-test-slave-lqrk krb5kdc[23317](info): commencing operation
krb5kdc: starting...
W20260926 03:43:09.507925 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:09 dist-test-slave-lqrk krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:09.784874 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:10.167291 23330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:10.167636 23330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:10.167778 23330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:10.175725 23330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:10.175880 23330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:10.175966 23330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:10.176049 23330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:10.176108 23330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:10.187176 23330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:10.188710 23330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:10.190128 23330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:10.202196 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:10.202521 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:10.202970 23338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:10.203356 23330 server_base.cc:1034] running on GCE node
I20260926 03:43:10.204221 23330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:10.205785 23330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:10.207142 23330 hybrid_clock.cc:648] HybridClock initialized: now 1790394190207054 us; error 36 us; skew 500 ppm
I20260926 03:43:10.207842 23330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:10 dist-test-slave-lqrk krb5kdc[23317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:10.217681 23330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:10.219587 23330 webserver.cc:460] Webserver started at http://127.0.0.1:44685/ using document root <none> and password file <none>
I20260926 03:43:10.220371 23330 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:10.220487 23330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:10.220780 23330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:10.224032 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "800c42b004ac4c11966887dc14ca1557"
format_stamp: "Formatted at 2026-09-26 03:43:10 on dist-test-slave-lqrk"
server_key: "0cc78e6c830aed06048831526c36589d"
server_key_iv: "05582739adc1965397b4ade4a8850c3f"
server_key_version: "encryptionkey@0"
I20260926 03:43:10.224870 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "800c42b004ac4c11966887dc14ca1557"
format_stamp: "Formatted at 2026-09-26 03:43:10 on dist-test-slave-lqrk"
server_key: "0cc78e6c830aed06048831526c36589d"
server_key_iv: "05582739adc1965397b4ade4a8850c3f"
server_key_version: "encryptionkey@0"
I20260926 03:43:10.230568 23330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:43:10.234532 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:10.236240 23330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:43:10.236435 23330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "800c42b004ac4c11966887dc14ca1557"
format_stamp: "Formatted at 2026-09-26 03:43:10 on dist-test-slave-lqrk"
server_key: "0cc78e6c830aed06048831526c36589d"
server_key_iv: "05582739adc1965397b4ade4a8850c3f"
server_key_version: "encryptionkey@0"
I20260926 03:43:10.236653 23330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:10.250690 23330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:10.256146 23330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:10.256496 23330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:10.280417 23330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39707
I20260926 03:43:10.280435 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39707 every 8 connection(s)
I20260926 03:43:10.282236 23330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:10.283293 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23330
I20260926 03:43:10.283527 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:10.284022 1910 external_mini_cluster.cc:1428] Setting key 26eda446a920c72c2ea21b78461c72b7
I20260926 03:43:10.288095 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:43:10 dist-test-slave-lqrk krb5kdc[23317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:43:10.299934 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557: Bootstrap starting.
I20260926 03:43:10.303774 23397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:10.304883 23397 log.cc:824] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:10.308285 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557: No bootstrap required, opened a new log
I20260926 03:43:10.309224 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:10.286123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:59063 (local address 127.0.0.1:39707)
0926 03:43:10.287196 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:10.287208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:10.287341 (+ 133us) server_negotiation.cc:407] Connection header received
0926 03:43:10.289762 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:10.289788 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:10.290267 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:10.290695 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:10.292322 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:10.293820 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:10.295146 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:10.295915 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:10.296111 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:10.300400 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:10.300409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:10.300426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:10.300490 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:10.304019 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:10.304914 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:10.304921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:10.304928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:10.305150 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:10.305788 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:10.305792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:10.305795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:10.306505 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:10.306856 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:10.307803 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 03:43:10.308167 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":391,"threads_started":1}
I20260926 03:43:10.314628 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:10.317992 23397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260926 03:43:10.318442 23397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:10.318584 23397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800c42b004ac4c11966887dc14ca1557, State: Initialized, Role: FOLLOWER
I20260926 03:43:10.319199 23397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260926 03:43:10.319474 23397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:10.319742 23397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:10.320248 23397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:10.323329 23397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260926 03:43:10.324472 23397 leader_election.cc:302] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800c42b004ac4c11966887dc14ca1557; no voters:
I20260926 03:43:10.325405 23397 leader_election.cc:288] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:10.325526 23407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:10.326295 23407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [term 1 LEADER]: Becoming Leader. State: Replica: 800c42b004ac4c11966887dc14ca1557, State: Running, Role: LEADER
I20260926 03:43:10.327313 23407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260926 03:43:10.329375 23397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:10.330829 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "800c42b004ac4c11966887dc14ca1557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } } }
I20260926 03:43:10.331293 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:10.332098 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800c42b004ac4c11966887dc14ca1557. Latest consensus state: current_term: 1 leader_uuid: "800c42b004ac4c11966887dc14ca1557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800c42b004ac4c11966887dc14ca1557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } } }
I20260926 03:43:10.332448 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:10.339637 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:10.319580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:51481 (local address 127.0.0.1:39707)
0926 03:43:10.320151 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:10.320158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:10.320286 (+ 128us) server_negotiation.cc:407] Connection header received
0926 03:43:10.320518 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:10.320527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:10.320605 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:10.320696 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:10.322264 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:10.323079 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:10.324584 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:10.325069 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:10.325210 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:10.329545 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:10.329553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:10.329558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:10.329611 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:10.334586 (+ 4975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:10.336385 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:10.336393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:10.336397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:10.336626 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:10.337321 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:10.337328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:10.337332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:10.337790 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:10.338114 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:10.338917 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 03:43:10.339172 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260926 03:43:10.343259 23412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:10.349545 23412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:10.363961 23412 catalog_manager.cc:1347] Generated new cluster ID: 8fe157938ead4d2bb6a98efb5ef4ad63
I20260926 03:43:10.364140 23412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:10.391978 23412 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:10.394049 23412 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:10.415773 23412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800c42b004ac4c11966887dc14ca1557: Generated new TSK 0
I20260926 03:43:10.417452 23412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:10.436668 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23330
I20260926 03:43:10.438930 23392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:10.540323 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23330
W20260926 03:43:10.555707 23315 connection.cc:535] client connection to 127.0.0.1:39707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39707): Connection reset by peer (error 104)
2026-09-26T03:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:10 dist-test-slave-lqrk krb5kdc[23429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:10 dist-test-slave-lqrk krb5kdc[23429](info): set up 2 sockets
Sep 26 03:43:10 dist-test-slave-lqrk krb5kdc[23429](info): commencing operation
krb5kdc: starting...
W20260926 03:43:11.631992 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:11 dist-test-slave-lqrk krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:11.952610 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:12.381038 23442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:12.381480 23442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:12.381635 23442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:12.389985 23442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:12.390180 23442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:12.390256 23442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:12.390344 23442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:12.390410 23442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:12.403751 23442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:12.405632 23442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:12.407361 23442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:12.422432 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:12.422415 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:12.422461 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:12.423969 23442 server_base.cc:1034] running on GCE node
I20260926 03:43:12.425385 23442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:12.428023 23442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:12.431540 23442 hybrid_clock.cc:648] HybridClock initialized: now 1790394192431426 us; error 74 us; skew 500 ppm
I20260926 03:43:12.432319 23442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:12 dist-test-slave-lqrk krb5kdc[23429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:12.437441 23442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:12.439729 23442 webserver.cc:460] Webserver started at http://127.0.0.1:34801/ using document root <none> and password file <none>
I20260926 03:43:12.440647 23442 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:12.440778 23442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:12.441085 23442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:12.445240 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "619dbc39dfb541f69b8d9b297b790fd1"
format_stamp: "Formatted at 2026-09-26 03:43:12 on dist-test-slave-lqrk"
server_key: "284efba81b1c06775296a38ebb3777b2"
server_key_iv: "a92ceda3f228bad18e7c084a35205944"
server_key_version: "encryptionkey@0"
I20260926 03:43:12.446305 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "619dbc39dfb541f69b8d9b297b790fd1"
format_stamp: "Formatted at 2026-09-26 03:43:12 on dist-test-slave-lqrk"
server_key: "284efba81b1c06775296a38ebb3777b2"
server_key_iv: "a92ceda3f228bad18e7c084a35205944"
server_key_version: "encryptionkey@0"
I20260926 03:43:12.453536 23442 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 03:43:12.458595 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:12.460785 23442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:43:12.461050 23442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "619dbc39dfb541f69b8d9b297b790fd1"
format_stamp: "Formatted at 2026-09-26 03:43:12 on dist-test-slave-lqrk"
server_key: "284efba81b1c06775296a38ebb3777b2"
server_key_iv: "a92ceda3f228bad18e7c084a35205944"
server_key_version: "encryptionkey@0"
I20260926 03:43:12.461313 23442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:12.523815 23442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:12.530020 23442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:12.530407 23442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:12.553958 23442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34231
I20260926 03:43:12.553983 23508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34231 every 8 connection(s)
I20260926 03:43:12.555881 23442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:12.557719 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23442
I20260926 03:43:12.557926 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:12.558485 1910 external_mini_cluster.cc:1428] Setting key 0264d18231362c5d78bc89a4911d5d98
I20260926 03:43:12.561425 23509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:12.576349 23509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1: Bootstrap starting.
I20260926 03:43:12.576546 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:12.560923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:40407 (local address 127.0.0.1:34231)
0926 03:43:12.561931 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:12.561947 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:12.562002 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:43:12.564032 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:12.564064 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:12.564655 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:12.565146 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:12.565998 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:12.568043 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:12.570075 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:12.571131 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:12.571349 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:12.573021 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:12.573035 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:12.573058 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:12.574106 (+ 1048us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:12.574224 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:12.574266 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:12.574455 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:12.574800 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 03:43:12.575120 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":440,"threads_started":1}
I20260926 03:43:12.580475 23509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:12.581748 23509 log.cc:824] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:12.582417 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:12.585505 23509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1: No bootstrap required, opened a new log
I20260926 03:43:12.594813 23509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260926 03:43:12.595266 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:12.588017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:41935 (local address 127.0.0.1:34231)
0926 03:43:12.588249 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:12.588257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:12.588757 (+ 500us) server_negotiation.cc:407] Connection header received
0926 03:43:12.589006 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:12.589011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:12.589097 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:12.589205 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:12.589827 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:12.590759 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:12.592330 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:12.592896 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:12.593043 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:12.594393 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:12.594398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:12.594402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:12.594427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:12.594501 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:12.594522 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:12.594630 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:12.594834 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 03:43:12.594906 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:43:12.595458 23509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:12.595600 23509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619dbc39dfb541f69b8d9b297b790fd1, State: Initialized, Role: FOLLOWER
I20260926 03:43:12.596367 23509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260926 03:43:12.596695 23509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:12.596851 23509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:12.597066 23509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:12.599694 23509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260926 03:43:12.600301 23509 leader_election.cc:302] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619dbc39dfb541f69b8d9b297b790fd1; no voters:
I20260926 03:43:12.601022 23509 leader_election.cc:288] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:12.601253 23519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:12.602246 23519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 619dbc39dfb541f69b8d9b297b790fd1, State: Running, Role: LEADER
I20260926 03:43:12.603065 23519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } }
I20260926 03:43:12.603919 23509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:12.608383 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619dbc39dfb541f69b8d9b297b790fd1. Latest consensus state: current_term: 1 leader_uuid: "619dbc39dfb541f69b8d9b297b790fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } } }
I20260926 03:43:12.609006 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619dbc39dfb541f69b8d9b297b790fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619dbc39dfb541f69b8d9b297b790fd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34231 } } }
I20260926 03:43:12.609408 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:12.609468 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:12.613018 23526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:12.620141 23526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:12.632241 23526 catalog_manager.cc:1347] Generated new cluster ID: be4cd768f6604cdd965ed3f1a9cbae55
I20260926 03:43:12.632406 23526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:12.645751 23526 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:12.648051 23526 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:12.657645 23526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619dbc39dfb541f69b8d9b297b790fd1: Generated new TSK 0
I20260926 03:43:12.658789 23526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:12.676483 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23442
W20260926 03:43:12.677783 23504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:40407
W20260926 03:43:12.678726 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23442: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:12.678877 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23442
W20260926 03:43:12.696004 23427 connection.cc:535] client connection to 127.0.0.1:34231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34231): Connection reset by peer (error 104)
2026-09-26T03:43:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:12 dist-test-slave-lqrk krb5kdc[23540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:12 dist-test-slave-lqrk krb5kdc[23540](info): set up 2 sockets
Sep 26 03:43:12 dist-test-slave-lqrk krb5kdc[23540](info): commencing operation
krb5kdc: starting...
W20260926 03:43:13.744132 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:13 dist-test-slave-lqrk krb5kdc[23540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:43:14.060608 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:14.466773 23553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:14.467105 23553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:14.467222 23553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:14.476502 23553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:14.476739 23553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:14.476800 23553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:14.476912 23553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:14.476984 23553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:14.488881 23553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:14.490469 23553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:14.492000 23553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:14.507524 23559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:14.507596 23561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:14.507606 23558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:14.513634 23553 server_base.cc:1034] running on GCE node
I20260926 03:43:14.514900 23553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:14.517158 23553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:14.518471 23553 hybrid_clock.cc:648] HybridClock initialized: now 1790394194518394 us; error 62 us; skew 500 ppm
I20260926 03:43:14.519039 23553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:14 dist-test-slave-lqrk krb5kdc[23540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:14.523263 23553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:43:14.525352 23553 webserver.cc:460] Webserver started at http://127.0.0.1:38175/ using document root <none> and password file <none>
I20260926 03:43:14.526185 23553 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:14.526314 23553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:14.526635 23553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:14.530198 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "bea4f7339dbb468a921278d880ad6485"
format_stamp: "Formatted at 2026-09-26 03:43:14 on dist-test-slave-lqrk"
server_key: "057cad2ae13f509936bd53e4454258fd"
server_key_iv: "6c83090c65a2ded13a72f59608a00ea9"
server_key_version: "encryptionkey@0"
I20260926 03:43:14.531023 23553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "bea4f7339dbb468a921278d880ad6485"
format_stamp: "Formatted at 2026-09-26 03:43:14 on dist-test-slave-lqrk"
server_key: "057cad2ae13f509936bd53e4454258fd"
server_key_iv: "6c83090c65a2ded13a72f59608a00ea9"
server_key_version: "encryptionkey@0"
I20260926 03:43:14.537631 23553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260926 03:43:14.542328 23568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:14.544355 23553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 03:43:14.544682 23553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "bea4f7339dbb468a921278d880ad6485"
format_stamp: "Formatted at 2026-09-26 03:43:14 on dist-test-slave-lqrk"
server_key: "057cad2ae13f509936bd53e4454258fd"
server_key_iv: "6c83090c65a2ded13a72f59608a00ea9"
server_key_version: "encryptionkey@0"
I20260926 03:43:14.544938 23553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:14.552597 23553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:14.558501 23553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:14.558866 23553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:14.582914 23553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33787
I20260926 03:43:14.583035 23619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33787 every 8 connection(s)
I20260926 03:43:14.584997 23553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:14.585465 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23553
I20260926 03:43:14.585681 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:14.586387 1910 external_mini_cluster.cc:1428] Setting key 2f568700cb157ab31c9779ce6f6872d7
I20260926 03:43:14.591177 23620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:14.596854 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:14.589082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:60929 (local address 127.0.0.1:33787)
0926 03:43:14.590164 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:14.590179 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:14.590236 (+ 57us) server_negotiation.cc:407] Connection header received
0926 03:43:14.592216 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:14.592249 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:14.592718 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:14.593165 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:14.594440 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:14.594448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:14.594465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:14.594484 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:14.594699 (+ 215us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:14.594746 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:14.595136 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:14.595317 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 03:43:14.595693 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":459,"threads_started":1}
I20260926 03:43:14.603765 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:14.609205 23620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485: Bootstrap starting.
I20260926 03:43:14.613018 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:14.609094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:49301 (local address 127.0.0.1:33787)
0926 03:43:14.609337 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:14.609343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:14.609863 (+ 520us) server_negotiation.cc:407] Connection header received
0926 03:43:14.610188 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:14.610193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:14.610262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:14.610386 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:14.611874 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:14.611879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:14.611883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:14.611906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:14.611983 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:14.612007 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:14.612161 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:14.612377 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:43:14.612466 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 03:43:14.615860 23620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:14.617805 23620 log.cc:824] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:14.622332 23620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485: No bootstrap required, opened a new log
I20260926 03:43:14.635509 23620 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260926 03:43:14.636343 23620 raft_consensus.cc:374] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:14.636488 23620 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea4f7339dbb468a921278d880ad6485, State: Initialized, Role: FOLLOWER
I20260926 03:43:14.637465 23620 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260926 03:43:14.637830 23620 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:14.638008 23620 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:14.638212 23620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:14.641623 23620 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260926 03:43:14.642450 23620 leader_election.cc:302] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea4f7339dbb468a921278d880ad6485; no voters:
I20260926 03:43:14.643339 23620 leader_election.cc:288] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:14.643746 23630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:14.644594 23630 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [term 1 LEADER]: Becoming Leader. State: Replica: bea4f7339dbb468a921278d880ad6485, State: Running, Role: LEADER
I20260926 03:43:14.645256 23630 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260926 03:43:14.646740 23620 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:14.649052 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea4f7339dbb468a921278d880ad6485. Latest consensus state: current_term: 1 leader_uuid: "bea4f7339dbb468a921278d880ad6485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260926 03:43:14.649170 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bea4f7339dbb468a921278d880ad6485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea4f7339dbb468a921278d880ad6485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260926 03:43:14.649796 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:14.649798 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:14.653093 23636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:14.661123 23636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:14.675252 23636 catalog_manager.cc:1347] Generated new cluster ID: 2fa632aa70bb4bc384872f6ec09f66a6
I20260926 03:43:14.675457 23636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:14.693584 23636 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:14.694088 23636 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:14.703821 23636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bea4f7339dbb468a921278d880ad6485: Generated new TSK 0
I20260926 03:43:14.705094 23636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:14.716990 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23553
W20260926 03:43:14.718321 23615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:60929
W20260926 03:43:14.719223 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:14.719431 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23553
2026-09-26T03:43:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:14 dist-test-slave-lqrk krb5kdc[23651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:14 dist-test-slave-lqrk krb5kdc[23651](info): set up 2 sockets
Sep 26 03:43:14 dist-test-slave-lqrk krb5kdc[23651](info): commencing operation
krb5kdc: starting...
W20260926 03:43:15.784409 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:15 dist-test-slave-lqrk krb5kdc[23651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:16.118084 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33427
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:16.517256 23664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:16.517593 23664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:16.517727 23664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:16.525918 23664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:16.526060 23664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:16.526108 23664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:16.526178 23664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:16.526247 23664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:16.536680 23664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:16.538061 23664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:16.539423 23664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:16.552664 23672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:16.552798 23670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:16.553287 23669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:16.554268 23664 server_base.cc:1034] running on GCE node
I20260926 03:43:16.555192 23664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:16.556900 23664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:16.558198 23664 hybrid_clock.cc:648] HybridClock initialized: now 1790394196558136 us; error 74 us; skew 500 ppm
I20260926 03:43:16.558665 23664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:16 dist-test-slave-lqrk krb5kdc[23651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:16.571303 23664 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:16.573731 23664 webserver.cc:460] Webserver started at http://127.1.221.190:44715/ using document root <none> and password file <none>
I20260926 03:43:16.574931 23664 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:16.575282 23664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:16.576027 23664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:16.581146 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "a37e1e6c1dd84c8687741e94a604783e"
format_stamp: "Formatted at 2026-09-26 03:43:16 on dist-test-slave-lqrk"
server_key: "77c0ae7f458b59164d4a44f314f17d01"
server_key_iv: "c90eb329b831cecc441ca3dd757d0ff1"
server_key_version: "encryptionkey@0"
I20260926 03:43:16.582434 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a37e1e6c1dd84c8687741e94a604783e"
format_stamp: "Formatted at 2026-09-26 03:43:16 on dist-test-slave-lqrk"
server_key: "77c0ae7f458b59164d4a44f314f17d01"
server_key_iv: "c90eb329b831cecc441ca3dd757d0ff1"
server_key_version: "encryptionkey@0"
I20260926 03:43:16.591159 23664 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260926 03:43:16.598925 23679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:16.601812 23664 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260926 03:43:16.602296 23664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "a37e1e6c1dd84c8687741e94a604783e"
format_stamp: "Formatted at 2026-09-26 03:43:16 on dist-test-slave-lqrk"
server_key: "77c0ae7f458b59164d4a44f314f17d01"
server_key_iv: "c90eb329b831cecc441ca3dd757d0ff1"
server_key_version: "encryptionkey@0"
I20260926 03:43:16.602644 23664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:16.621500 23664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:16.627645 23664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:16.627969 23664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:16.652027 23664 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33427
I20260926 03:43:16.652045 23730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33427 every 8 connection(s)
I20260926 03:43:16.654286 23664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:16.658800 23731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:16.663990 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23664
I20260926 03:43:16.664307 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:16.664992 1910 external_mini_cluster.cc:1428] Setting key 5dea84556fa1733c67606ed93edb572b
I20260926 03:43:16.673046 23731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e: Bootstrap starting.
I20260926 03:43:16.678176 23731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:16.679967 23731 log.cc:824] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:16.683750 23731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e: No bootstrap required, opened a new log
Sep 26 03:43:16 dist-test-slave-lqrk krb5kdc[23651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:43:16.693408 23731 raft_consensus.cc:348] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } }
I20260926 03:43:16.694144 23731 raft_consensus.cc:374] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:16.694274 23731 raft_consensus.cc:729] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37e1e6c1dd84c8687741e94a604783e, State: Initialized, Role: FOLLOWER
I20260926 03:43:16.695047 23731 consensus_queue.cc:260] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } }
I20260926 03:43:16.695420 23731 raft_consensus.cc:388] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:16.695601 23731 raft_consensus.cc:482] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:16.695813 23731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:16.699047 23731 raft_consensus.cc:504] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } }
I20260926 03:43:16.699698 23731 leader_election.cc:302] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a37e1e6c1dd84c8687741e94a604783e; no voters:
I20260926 03:43:16.700457 23731 leader_election.cc:288] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:16.700728 23736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:16.701697 23736 raft_consensus.cc:686] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [term 1 LEADER]: Becoming Leader. State: Replica: a37e1e6c1dd84c8687741e94a604783e, State: Running, Role: LEADER
I20260926 03:43:16.702625 23736 consensus_queue.cc:237] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } }
I20260926 03:43:16.703366 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:16.668460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42319 (local address 127.1.221.190:33427)
0926 03:43:16.669742 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:16.669765 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:16.669843 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:43:16.672902 (+ 3059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:16.672957 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:16.673850 (+ 893us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:16.676975 (+ 3125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:16.677939 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:16.680424 (+ 2485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:16.682802 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:16.684300 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:16.684610 (+ 310us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:16.690933 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:16.690953 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:16.690984 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:16.691061 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:16.696050 (+ 4989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:16.697581 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:16.697593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:16.697616 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:16.697928 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:16.698628 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:16.698633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:16.698635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:16.699659 (+ 1024us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:16.700091 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:16.700798 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 03:43:16.701364 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":702,"threads_started":1}
I20260926 03:43:16.704219 23731 sys_catalog.cc:564] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:16.709092 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a37e1e6c1dd84c8687741e94a604783e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } } }
I20260926 03:43:16.709674 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:16.710358 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:16.711616 23737 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a37e1e6c1dd84c8687741e94a604783e. Latest consensus state: current_term: 1 leader_uuid: "a37e1e6c1dd84c8687741e94a604783e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37e1e6c1dd84c8687741e94a604783e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33427 } } }
I20260926 03:43:16.712003 23737 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:16.712481 23745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:16.719686 23745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:16.741134 23745 catalog_manager.cc:1347] Generated new cluster ID: c157a52646434217a763f478b6b9bab5
I20260926 03:43:16.741294 23745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:16.757397 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:16.727196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47269 (local address 127.1.221.190:33427)
0926 03:43:16.727458 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:16.727465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:16.734936 (+ 7471us) server_negotiation.cc:407] Connection header received
0926 03:43:16.735173 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:16.735181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:16.735307 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:16.735476 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:16.738123 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:16.739638 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:16.741802 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:16.742606 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:16.742723 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:16.747210 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:16.747219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:16.747226 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:16.747287 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:16.751933 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:16.753400 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:16.753408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:16.753413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:16.753920 (+ 507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:16.754805 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:16.754813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:16.754817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:16.755326 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:16.755800 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:16.756633 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 03:43:16.756907 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":85}
I20260926 03:43:16.762642 23745 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:16.764592 23745 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:16.771829 23745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a37e1e6c1dd84c8687741e94a604783e: Generated new TSK 0
I20260926 03:43:16.772974 23745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:16.779662 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23664
I20260926 03:43:16.780889 23726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:16.875934 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23664
W20260926 03:43:16.892441 23649 connection.cc:535] client connection to 127.1.221.190:33427 recv error: Network error: failed to read from TLS socket (remote: 127.1.221.190:33427): Connection reset by peer (error 104)
2026-09-26T03:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:16 dist-test-slave-lqrk krb5kdc[23763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:16 dist-test-slave-lqrk krb5kdc[23763](info): set up 2 sockets
Sep 26 03:43:16 dist-test-slave-lqrk krb5kdc[23763](info): commencing operation
krb5kdc: starting...
W20260926 03:43:17.936064 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:18 dist-test-slave-lqrk krb5kdc[23763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:18.248377 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35235
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:18.690798 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:18.691216 23776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:18.691372 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:18.699947 23776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:18.700119 23776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:18.700179 23776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:18.700361 23776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:18.700434 23776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:18.712397 23776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:18.714192 23776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:18.715889 23776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:18.731218 23781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:18.731251 23782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:18.731364 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:18.732414 23776 server_base.cc:1034] running on GCE node
I20260926 03:43:18.733562 23776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:18.735819 23776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:18.737179 23776 hybrid_clock.cc:648] HybridClock initialized: now 1790394198737116 us; error 81 us; skew 500 ppm
I20260926 03:43:18.737856 23776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:18 dist-test-slave-lqrk krb5kdc[23763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:18.742473 23776 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:18.744865 23776 webserver.cc:460] Webserver started at http://127.1.221.190:39117/ using document root <none> and password file <none>
I20260926 03:43:18.745893 23776 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:18.746052 23776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:18.746462 23776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:18.750648 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f"
format_stamp: "Formatted at 2026-09-26 03:43:18 on dist-test-slave-lqrk"
server_key: "b50f8ad68820601df1a6cf14411ab857"
server_key_iv: "3a70cc5143a4984d2ec7cb74235d9505"
server_key_version: "encryptionkey@0"
I20260926 03:43:18.751735 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f"
format_stamp: "Formatted at 2026-09-26 03:43:18 on dist-test-slave-lqrk"
server_key: "b50f8ad68820601df1a6cf14411ab857"
server_key_iv: "3a70cc5143a4984d2ec7cb74235d9505"
server_key_version: "encryptionkey@0"
I20260926 03:43:18.759346 23776 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260926 03:43:18.764964 23791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:18.767207 23776 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:43:18.767534 23776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f"
format_stamp: "Formatted at 2026-09-26 03:43:18 on dist-test-slave-lqrk"
server_key: "b50f8ad68820601df1a6cf14411ab857"
server_key_iv: "3a70cc5143a4984d2ec7cb74235d9505"
server_key_version: "encryptionkey@0"
I20260926 03:43:18.767827 23776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:18.805405 23776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:18.811060 23776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:18.811434 23776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:18.833088 23776 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35235
I20260926 03:43:18.833120 23842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35235 every 8 connection(s)
I20260926 03:43:18.834787 23776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:18.840265 23843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:18.842393 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23776
I20260926 03:43:18.842667 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:18.843132 1910 external_mini_cluster.cc:1428] Setting key 9f25a0fca20a4a37db8ce53e6b30927d
I20260926 03:43:18.858299 23843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f: Bootstrap starting.
I20260926 03:43:18.860508 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:18.845574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40241 (local address 127.1.221.190:35235)
0926 03:43:18.846799 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:18.846821 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:18.846885 (+ 64us) server_negotiation.cc:407] Connection header received
0926 03:43:18.849343 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:18.849402 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:18.850084 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:18.850648 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:18.851772 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:18.853375 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:18.854938 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:18.855854 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:18.856071 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:18.857683 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:18.857693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:18.857715 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:18.858374 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:18.858470 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:18.858508 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:18.858688 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:18.859004 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 03:43:18.859206 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":501,"threads_started":1}
I20260926 03:43:18.863134 23843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:18.864781 23843 log.cc:824] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:18.866325 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:18.868990 23843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f: No bootstrap required, opened a new log
I20260926 03:43:18.880249 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:18.871672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46205 (local address 127.1.221.190:35235)
0926 03:43:18.871879 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:18.871883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:18.872352 (+ 469us) server_negotiation.cc:407] Connection header received
0926 03:43:18.872681 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:18.872687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:18.872782 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:18.872922 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:18.873704 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:18.874599 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:18.876869 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:18.877502 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:18.877650 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:18.879314 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:18.879319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:18.879323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:18.879345 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:18.879431 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:18.879455 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:18.879577 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:18.879810 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 03:43:18.879866 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:43:18.880810 23843 raft_consensus.cc:348] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } }
I20260926 03:43:18.881357 23843 raft_consensus.cc:374] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:18.881477 23843 raft_consensus.cc:729] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4e35fbcd974fd0bdee5c6dac5b3b1f, State: Initialized, Role: FOLLOWER
I20260926 03:43:18.882212 23843 consensus_queue.cc:260] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } }
I20260926 03:43:18.882504 23843 raft_consensus.cc:388] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:18.882649 23843 raft_consensus.cc:482] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:18.882826 23843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:18.885682 23843 raft_consensus.cc:504] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } }
I20260926 03:43:18.886361 23843 leader_election.cc:302] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db4e35fbcd974fd0bdee5c6dac5b3b1f; no voters:
I20260926 03:43:18.887135 23843 leader_election.cc:288] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:18.887409 23853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:18.888424 23853 raft_consensus.cc:686] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [term 1 LEADER]: Becoming Leader. State: Replica: db4e35fbcd974fd0bdee5c6dac5b3b1f, State: Running, Role: LEADER
I20260926 03:43:18.889331 23853 consensus_queue.cc:237] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } }
I20260926 03:43:18.889956 23843 sys_catalog.cc:564] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:18.893558 23855 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader db4e35fbcd974fd0bdee5c6dac5b3b1f. Latest consensus state: current_term: 1 leader_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } } }
I20260926 03:43:18.893991 23855 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:18.894339 23854 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4e35fbcd974fd0bdee5c6dac5b3b1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35235 } } }
I20260926 03:43:18.894812 23854 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:18.900542 23861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:18.906198 23861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:18.918154 23861 catalog_manager.cc:1347] Generated new cluster ID: 7ab49ebfb0f541d1b5333d34eb9f491b
I20260926 03:43:18.918296 23861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:18.955113 23861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:18.957561 23861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:18.970801 23861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db4e35fbcd974fd0bdee5c6dac5b3b1f: Generated new TSK 0
I20260926 03:43:18.972358 23861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:18.994596 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23776
W20260926 03:43:18.996312 23838 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40241
W20260926 03:43:18.997210 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:18.997366 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23776
W20260926 03:43:19.014117 23761 connection.cc:535] client connection to 127.1.221.190:35235 recv error: Network error: failed to read from TLS socket (remote: 127.1.221.190:35235): Connection reset by peer (error 104)
2026-09-26T03:43:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:19 dist-test-slave-lqrk krb5kdc[23874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:19 dist-test-slave-lqrk krb5kdc[23874](info): set up 2 sockets
Sep 26 03:43:19 dist-test-slave-lqrk krb5kdc[23874](info): commencing operation
krb5kdc: starting...
W20260926 03:43:20.064417 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:20 dist-test-slave-lqrk krb5kdc[23874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:20.379026 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46107
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:46107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:20.810766 23887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:20.811211 23887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:20.811347 23887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:20.820489 23887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:20.820690 23887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:20.820786 23887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:20.820894 23887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:20.820962 23887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:20.834671 23887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:46107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:20.836355 23887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:20.837877 23887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:20.852496 23893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:20.852612 23892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:20.852574 23895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:20.853684 23887 server_base.cc:1034] running on GCE node
I20260926 03:43:20.854594 23887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:20.856488 23887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:20.857784 23887 hybrid_clock.cc:648] HybridClock initialized: now 1790394200857725 us; error 56 us; skew 500 ppm
I20260926 03:43:20.858244 23887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:20 dist-test-slave-lqrk krb5kdc[23874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:20.861967 23887 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:20.863993 23887 webserver.cc:460] Webserver started at http://127.1.221.190:40969/ using document root <none> and password file <none>
I20260926 03:43:20.864771 23887 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:20.864900 23887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:20.865235 23887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:20.868901 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "da886501cc7c4bd386c33a0f271ddf91"
format_stamp: "Formatted at 2026-09-26 03:43:20 on dist-test-slave-lqrk"
server_key: "91cc32aa5af22663a3f977c1dbcb8530"
server_key_iv: "346b914f6bf190612996b96cf473de7a"
server_key_version: "encryptionkey@0"
I20260926 03:43:20.869805 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "da886501cc7c4bd386c33a0f271ddf91"
format_stamp: "Formatted at 2026-09-26 03:43:20 on dist-test-slave-lqrk"
server_key: "91cc32aa5af22663a3f977c1dbcb8530"
server_key_iv: "346b914f6bf190612996b96cf473de7a"
server_key_version: "encryptionkey@0"
I20260926 03:43:20.876374 23887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:43:20.881072 23902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:20.883095 23887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 03:43:20.883338 23887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "da886501cc7c4bd386c33a0f271ddf91"
format_stamp: "Formatted at 2026-09-26 03:43:20 on dist-test-slave-lqrk"
server_key: "91cc32aa5af22663a3f977c1dbcb8530"
server_key_iv: "346b914f6bf190612996b96cf473de7a"
server_key_version: "encryptionkey@0"
I20260926 03:43:20.883649 23887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:20.890291 23887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:20.895361 23887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:20.895768 23887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:20.916955 23887 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:46107
I20260926 03:43:20.917028 23953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:46107 every 8 connection(s)
I20260926 03:43:20.918584 23887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:20.922742 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23887
I20260926 03:43:20.922974 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:20.923667 1910 external_mini_cluster.cc:1428] Setting key bbe6188070d80c4989d35debf1e1af1a
I20260926 03:43:20.924245 23954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:20.934633 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:20.926122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33831 (local address 127.1.221.190:46107)
0926 03:43:20.927048 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:20.927062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:20.927112 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:43:20.929875 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:20.929908 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:20.930391 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:20.930822 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:20.932547 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:20.932556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:20.932574 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:20.932609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:20.932734 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:20.932765 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:20.933010 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:20.933158 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:43:20.933399 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":376,"threads_started":1}
I20260926 03:43:20.939143 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:20.941447 23954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91: Bootstrap starting.
I20260926 03:43:20.946894 23954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:20.948433 23954 log.cc:824] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:20.948724 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:20.945514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33321 (local address 127.1.221.190:46107)
0926 03:43:20.945724 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:20.945729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:20.946091 (+ 362us) server_negotiation.cc:407] Connection header received
0926 03:43:20.946466 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:20.946472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:20.946575 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:20.946751 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:20.947990 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:20.947995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:20.948000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:20.948019 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:20.948064 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:20.948081 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:20.948204 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:20.948298 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:43:20.948364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:43:20.952492 23954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91: No bootstrap required, opened a new log
I20260926 03:43:20.961930 23954 raft_consensus.cc:348] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } }
I20260926 03:43:20.962469 23954 raft_consensus.cc:374] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:20.962581 23954 raft_consensus.cc:729] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da886501cc7c4bd386c33a0f271ddf91, State: Initialized, Role: FOLLOWER
I20260926 03:43:20.963340 23954 consensus_queue.cc:260] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } }
I20260926 03:43:20.963675 23954 raft_consensus.cc:388] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:20.963836 23954 raft_consensus.cc:482] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:20.964003 23954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:20.967006 23954 raft_consensus.cc:504] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } }
I20260926 03:43:20.967803 23954 leader_election.cc:302] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da886501cc7c4bd386c33a0f271ddf91; no voters:
I20260926 03:43:20.968590 23954 leader_election.cc:288] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:20.968972 23964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:20.969887 23964 raft_consensus.cc:686] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [term 1 LEADER]: Becoming Leader. State: Replica: da886501cc7c4bd386c33a0f271ddf91, State: Running, Role: LEADER
I20260926 03:43:20.970610 23964 consensus_queue.cc:237] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } }
I20260926 03:43:20.971529 23954 sys_catalog.cc:564] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:20.974382 23966 sys_catalog.cc:455] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da886501cc7c4bd386c33a0f271ddf91. Latest consensus state: current_term: 1 leader_uuid: "da886501cc7c4bd386c33a0f271ddf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } } }
I20260926 03:43:20.974545 23965 sys_catalog.cc:455] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da886501cc7c4bd386c33a0f271ddf91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da886501cc7c4bd386c33a0f271ddf91" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46107 } } }
I20260926 03:43:20.975069 23966 sys_catalog.cc:458] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:20.975075 23965 sys_catalog.cc:458] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:20.978564 23971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:20.985836 23971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:20.997576 23971 catalog_manager.cc:1347] Generated new cluster ID: 3a12a48da6df41ccade9a50a51ec6049
I20260926 03:43:20.997746 23971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:21.019266 23971 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:21.020020 23971 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:21.032822 23971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da886501cc7c4bd386c33a0f271ddf91: Generated new TSK 0
I20260926 03:43:21.034198 23971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:21.053972 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23887
W20260926 03:43:21.056231 23949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33831
W20260926 03:43:21.057180 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:21.057407 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23887
2026-09-26T03:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:21 dist-test-slave-lqrk krb5kdc[23985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:21 dist-test-slave-lqrk krb5kdc[23985](info): set up 2 sockets
Sep 26 03:43:21 dist-test-slave-lqrk krb5kdc[23985](info): commencing operation
krb5kdc: starting...
W20260926 03:43:22.139959 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:22 dist-test-slave-lqrk krb5kdc[23985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:22.464251 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43625
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:22.885403 23998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:22.885768 23998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:22.885885 23998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:22.893925 23998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:22.894073 23998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:22.894135 23998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:22.894225 23998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:22.894284 23998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:22.904902 23998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:22.906634 23998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:22.908385 23998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:22.923220 24004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:22.923453 24006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:22.923975 24003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:22.925524 23998 server_base.cc:1034] running on GCE node
I20260926 03:43:22.926533 23998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:22.928334 23998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:22.929652 23998 hybrid_clock.cc:648] HybridClock initialized: now 1790394202929583 us; error 65 us; skew 500 ppm
I20260926 03:43:22.930119 23998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:22 dist-test-slave-lqrk krb5kdc[23985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394202, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:22.938462 23998 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:22.940603 23998 webserver.cc:460] Webserver started at http://127.1.221.190:43909/ using document root <none> and password file <none>
I20260926 03:43:22.941413 23998 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:22.941550 23998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:22.941903 23998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:22.945998 23998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "d0dbfef2b7fd456c9be008150ef1f4ea"
format_stamp: "Formatted at 2026-09-26 03:43:22 on dist-test-slave-lqrk"
server_key: "7ea6d907cd0374d53ab150a893f0a6a0"
server_key_iv: "418a80506ae87ef6c342c6eb326232cd"
server_key_version: "encryptionkey@0"
I20260926 03:43:22.946990 23998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "d0dbfef2b7fd456c9be008150ef1f4ea"
format_stamp: "Formatted at 2026-09-26 03:43:22 on dist-test-slave-lqrk"
server_key: "7ea6d907cd0374d53ab150a893f0a6a0"
server_key_iv: "418a80506ae87ef6c342c6eb326232cd"
server_key_version: "encryptionkey@0"
I20260926 03:43:22.954049 23998 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:43:22.958853 24013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:22.960860 23998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:43:22.961146 23998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "d0dbfef2b7fd456c9be008150ef1f4ea"
format_stamp: "Formatted at 2026-09-26 03:43:22 on dist-test-slave-lqrk"
server_key: "7ea6d907cd0374d53ab150a893f0a6a0"
server_key_iv: "418a80506ae87ef6c342c6eb326232cd"
server_key_version: "encryptionkey@0"
I20260926 03:43:22.961393 23998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:22.988519 23998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:22.994899 23998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:22.995352 23998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:23.019726 23998 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43625
I20260926 03:43:23.019779 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43625 every 8 connection(s)
I20260926 03:43:23.021373 23998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:23.026888 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:23.027536 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 23998
I20260926 03:43:23.027813 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:23.028461 1910 external_mini_cluster.cc:1428] Setting key 548cf32de7295eff109b7a82b9da8c8a
I20260926 03:43:23.044574 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea: Bootstrap starting.
Sep 26 03:43:23 dist-test-slave-lqrk krb5kdc[23985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:43:23.049470 24065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:23.051167 24065 log.cc:824] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:23.055688 24065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea: No bootstrap required, opened a new log
I20260926 03:43:23.057235 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:23.030946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35505 (local address 127.1.221.190:43625)
0926 03:43:23.032113 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:23.032131 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:23.032187 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:43:23.035121 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:23.035163 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:23.035847 (+ 684us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:23.036336 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:23.038337 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:23.039896 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:23.041556 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:23.042448 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:23.042677 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:23.047922 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:23.047932 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:23.047953 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:23.048006 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:23.051871 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:23.053035 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:23.053045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:23.053053 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:23.053278 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:23.053845 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:23.053850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:23.053852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:23.054512 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:23.054773 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:23.055438 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 03:43:23.055948 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":519,"threads_started":1}
I20260926 03:43:23.061924 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:23.066916 24065 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } }
I20260926 03:43:23.067690 24065 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:23.067838 24065 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfef2b7fd456c9be008150ef1f4ea, State: Initialized, Role: FOLLOWER
I20260926 03:43:23.068800 24065 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } }
I20260926 03:43:23.069136 24065 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:23.069314 24065 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:23.069541 24065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:23.073035 24065 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } }
I20260926 03:43:23.073885 24065 leader_election.cc:302] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dbfef2b7fd456c9be008150ef1f4ea; no voters:
I20260926 03:43:23.074698 24065 leader_election.cc:288] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:23.074900 24075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:23.076514 24075 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfef2b7fd456c9be008150ef1f4ea, State: Running, Role: LEADER
I20260926 03:43:23.077363 24075 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } }
I20260926 03:43:23.078123 24065 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:23.081135 24077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dbfef2b7fd456c9be008150ef1f4ea. Latest consensus state: current_term: 1 leader_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } } }
I20260926 03:43:23.082413 24077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:23.081763 24076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dbfef2b7fd456c9be008150ef1f4ea" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43625 } } }
I20260926 03:43:23.083738 24076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:23.087756 24080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:23.091790 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:23.068500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48473 (local address 127.1.221.190:43625)
0926 03:43:23.068838 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:23.068846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:23.069130 (+ 284us) server_negotiation.cc:407] Connection header received
0926 03:43:23.069363 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:23.069370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:23.069480 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:23.069663 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:23.071510 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:23.072684 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:23.074371 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:23.075281 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:23.075513 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:23.078872 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:23.078880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:23.078886 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:23.078948 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:23.087031 (+ 8083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:23.088281 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:23.088287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:23.088290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:23.088455 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:23.088958 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:23.088962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:23.088965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:23.089425 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:23.090918 (+ 1493us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:23.091096 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:43:23.091405 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 03:43:23.094798 24080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:23.110438 24080 catalog_manager.cc:1347] Generated new cluster ID: 02728e52c4494d52ac2d266e66da4390
I20260926 03:43:23.110589 24080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:23.139981 24080 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:23.141445 24080 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:23.155194 24080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0dbfef2b7fd456c9be008150ef1f4ea: Generated new TSK 0
I20260926 03:43:23.156384 24080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:23.176415 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 23998
I20260926 03:43:23.177624 24060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:23.276746 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 23998
2026-09-26T03:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:23 dist-test-slave-lqrk krb5kdc[24097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:23 dist-test-slave-lqrk krb5kdc[24097](info): set up 2 sockets
Sep 26 03:43:23 dist-test-slave-lqrk krb5kdc[24097](info): commencing operation
krb5kdc: starting...
W20260926 03:43:24.352077 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:24 dist-test-slave-lqrk krb5kdc[24097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:24.661270 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33809
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:25.062834 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:25.063174 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:25.063308 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:25.071448 24110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:25.071607 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:25.071830 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:25.071949 24110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:25.072017 24110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:25.082859 24110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:25.084335 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:25.085741 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:25.099748 24118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:25.099841 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:25.099800 24116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:25.101142 24110 server_base.cc:1034] running on GCE node
I20260926 03:43:25.102370 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:25.104431 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:25.105700 24110 hybrid_clock.cc:648] HybridClock initialized: now 1790394205105625 us; error 62 us; skew 500 ppm
I20260926 03:43:25.106194 24110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:25 dist-test-slave-lqrk krb5kdc[24097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394205, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:25.110405 24110 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:25.112475 24110 webserver.cc:460] Webserver started at http://127.1.221.190:40495/ using document root <none> and password file <none>
I20260926 03:43:25.113402 24110 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:25.113574 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:25.113924 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:25.117734 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d"
format_stamp: "Formatted at 2026-09-26 03:43:25 on dist-test-slave-lqrk"
server_key: "67da8aaa49eb4c14e90c89c3e21d25cc"
server_key_iv: "7166d6dc24e06096ea4f2e9401265917"
server_key_version: "encryptionkey@0"
I20260926 03:43:25.118711 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d"
format_stamp: "Formatted at 2026-09-26 03:43:25 on dist-test-slave-lqrk"
server_key: "67da8aaa49eb4c14e90c89c3e21d25cc"
server_key_iv: "7166d6dc24e06096ea4f2e9401265917"
server_key_version: "encryptionkey@0"
I20260926 03:43:25.125778 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 03:43:25.130709 24125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:25.132750 24110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:43:25.132972 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d"
format_stamp: "Formatted at 2026-09-26 03:43:25 on dist-test-slave-lqrk"
server_key: "67da8aaa49eb4c14e90c89c3e21d25cc"
server_key_iv: "7166d6dc24e06096ea4f2e9401265917"
server_key_version: "encryptionkey@0"
I20260926 03:43:25.133203 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:25.158427 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:25.163867 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:25.164165 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:25.185766 24110 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33809
I20260926 03:43:25.185822 24176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33809 every 8 connection(s)
I20260926 03:43:25.187552 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:25.192051 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:25.195604 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24110
I20260926 03:43:25.195842 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:25.196417 1910 external_mini_cluster.cc:1428] Setting key 4df0a08063c1663ec326a3e9c8370fe6
I20260926 03:43:25.206431 24177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d: Bootstrap starting.
I20260926 03:43:25.210353 24177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:25.211762 24177 log.cc:824] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:25.213249 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:25.199502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46727 (local address 127.1.221.190:33809)
0926 03:43:25.200489 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:25.200507 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:25.200571 (+ 64us) server_negotiation.cc:407] Connection header received
0926 03:43:25.202571 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:25.202605 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:25.203263 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:25.203833 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:25.204562 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:25.206034 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:25.207560 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:25.208450 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:25.208648 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:25.210087 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:25.210098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:25.210118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:25.210954 (+ 836us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:25.211058 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:25.211090 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:25.211215 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:25.211510 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:43:25.211866 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":388,"threads_started":1}
I20260926 03:43:25.215343 24177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d: No bootstrap required, opened a new log
I20260926 03:43:25.218163 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:25.224252 24177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } }
I20260926 03:43:25.224845 24177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:25.224970 24177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771bb0b2bb4b42a8997aa5c2337a1d3d, State: Initialized, Role: FOLLOWER
I20260926 03:43:25.225775 24177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } }
I20260926 03:43:25.226128 24177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:25.226296 24177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:25.226498 24177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:25.229579 24177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } }
I20260926 03:43:25.230234 24177 leader_election.cc:302] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771bb0b2bb4b42a8997aa5c2337a1d3d; no voters:
I20260926 03:43:25.231009 24177 leader_election.cc:288] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:25.231151 24187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:25.231725 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:25.223676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42471 (local address 127.1.221.190:33809)
0926 03:43:25.223922 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:25.223929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:25.224137 (+ 208us) server_negotiation.cc:407] Connection header received
0926 03:43:25.224359 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:25.224366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:25.224466 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:25.224632 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:25.225340 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:25.226474 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:25.228240 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:25.228984 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:25.229153 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:25.230706 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:25.230713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:25.230717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:25.230747 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:25.230840 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:25.230861 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:25.230965 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:25.231109 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:43:25.231244 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 03:43:25.232034 24187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [term 1 LEADER]: Becoming Leader. State: Replica: 771bb0b2bb4b42a8997aa5c2337a1d3d, State: Running, Role: LEADER
I20260926 03:43:25.232992 24187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } }
I20260926 03:43:25.235781 24187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771bb0b2bb4b42a8997aa5c2337a1d3d. Latest consensus state: current_term: 1 leader_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } } }
I20260926 03:43:25.236364 24187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:25.237488 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771bb0b2bb4b42a8997aa5c2337a1d3d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33809 } } }
I20260926 03:43:25.238194 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:25.240597 24177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:25.240846 24191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:25.247453 24191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:25.259948 24191 catalog_manager.cc:1347] Generated new cluster ID: ef5062033fda4660af707226e26758df
I20260926 03:43:25.260118 24191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:25.274847 24191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:25.276984 24191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:25.287017 24191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771bb0b2bb4b42a8997aa5c2337a1d3d: Generated new TSK 0
I20260926 03:43:25.288337 24191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:25.298642 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24110
W20260926 03:43:25.300251 24172 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46727
W20260926 03:43:25.301188 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:25.301411 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24110
2026-09-26T03:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:25 dist-test-slave-lqrk krb5kdc[24208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:25 dist-test-slave-lqrk krb5kdc[24208](info): set up 2 sockets
Sep 26 03:43:25 dist-test-slave-lqrk krb5kdc[24208](info): commencing operation
krb5kdc: starting...
W20260926 03:43:26.368119 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:26 dist-test-slave-lqrk krb5kdc[24208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:26.676195 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39945
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:27.078908 24221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:27.079216 24221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:27.079312 24221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:27.086964 24221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:27.087092 24221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:27.087139 24221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:27.087210 24221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:27.087296 24221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:27.097882 24221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:27.099406 24221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:27.100859 24221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:27.114701 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:27.114832 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:27.114797 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:27.115917 24221 server_base.cc:1034] running on GCE node
I20260926 03:43:27.117004 24221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:27.118965 24221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:27.120219 24221 hybrid_clock.cc:648] HybridClock initialized: now 1790394207120163 us; error 48 us; skew 500 ppm
I20260926 03:43:27.120685 24221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:27 dist-test-slave-lqrk krb5kdc[24208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:27.124570 24221 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:27.126494 24221 webserver.cc:460] Webserver started at http://127.1.221.190:41055/ using document root <none> and password file <none>
I20260926 03:43:27.127411 24221 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:27.127557 24221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:27.127923 24221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:27.131719 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "ca0285772e3a409994ec271e0d4d4aff"
format_stamp: "Formatted at 2026-09-26 03:43:27 on dist-test-slave-lqrk"
server_key: "c6042e72bd5bf3e6dbf7f02ba5b8fd5e"
server_key_iv: "94ebea8a6639afd6b40166ad2ab3bd8a"
server_key_version: "encryptionkey@0"
I20260926 03:43:27.132673 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "ca0285772e3a409994ec271e0d4d4aff"
format_stamp: "Formatted at 2026-09-26 03:43:27 on dist-test-slave-lqrk"
server_key: "c6042e72bd5bf3e6dbf7f02ba5b8fd5e"
server_key_iv: "94ebea8a6639afd6b40166ad2ab3bd8a"
server_key_version: "encryptionkey@0"
I20260926 03:43:27.139309 24221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 03:43:27.143947 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:27.145823 24221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:43:27.146052 24221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "ca0285772e3a409994ec271e0d4d4aff"
format_stamp: "Formatted at 2026-09-26 03:43:27 on dist-test-slave-lqrk"
server_key: "c6042e72bd5bf3e6dbf7f02ba5b8fd5e"
server_key_iv: "94ebea8a6639afd6b40166ad2ab3bd8a"
server_key_version: "encryptionkey@0"
I20260926 03:43:27.146296 24221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:27.153628 24221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:27.158792 24221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:27.159153 24221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:27.180747 24221 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39945
I20260926 03:43:27.180791 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39945 every 8 connection(s)
I20260926 03:43:27.182570 24221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:27.187197 24288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:27.190866 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24221
I20260926 03:43:27.191071 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:27.191638 1910 external_mini_cluster.cc:1428] Setting key ec2e04589771d9ccf1ddda018f92d774
I20260926 03:43:27.201627 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:27.194264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38693 (local address 127.1.221.190:39945)
0926 03:43:27.195306 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:27.195328 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:27.195412 (+ 84us) server_negotiation.cc:407] Connection header received
0926 03:43:27.197409 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:27.197440 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:27.197944 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:27.198409 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:27.199727 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:27.199735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:27.199751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:27.199769 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:27.199847 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:27.199876 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:27.200071 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:27.200183 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 03:43:27.200412 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":482,"threads_started":1}
I20260926 03:43:27.201885 24288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff: Bootstrap starting.
I20260926 03:43:27.205633 24288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:27.206240 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:27.206905 24288 log.cc:824] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:27.210703 24288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff: No bootstrap required, opened a new log
I20260926 03:43:27.214815 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:27.211447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60105 (local address 127.1.221.190:39945)
0926 03:43:27.211680 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:27.211689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:27.212289 (+ 600us) server_negotiation.cc:407] Connection header received
0926 03:43:27.212504 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:27.212508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:27.212581 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:27.212696 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:27.214018 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:27.214027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:27.214031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:43:27.214053 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:27.214110 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:43:27.214129 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:27.214250 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:27.214397 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:43:27.214462 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:43:27.219720 24288 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } }
I20260926 03:43:27.220147 24288 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:43:27.220240 24288 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0285772e3a409994ec271e0d4d4aff, State: Initialized, Role: FOLLOWER
I20260926 03:43:27.220934 24288 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } }
I20260926 03:43:27.221257 24288 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:43:27.221390 24288 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:43:27.221549 24288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:27.223912 24288 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } }
I20260926 03:43:27.224457 24288 leader_election.cc:302] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0285772e3a409994ec271e0d4d4aff; no voters:
I20260926 03:43:27.225142 24288 leader_election.cc:288] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:43:27.225405 24298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:27.227018 24298 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [term 1 LEADER]: Becoming Leader. State: Replica: ca0285772e3a409994ec271e0d4d4aff, State: Running, Role: LEADER
I20260926 03:43:27.227864 24298 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } }
I20260926 03:43:27.228057 24288 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:27.231974 24300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0285772e3a409994ec271e0d4d4aff. Latest consensus state: current_term: 1 leader_uuid: "ca0285772e3a409994ec271e0d4d4aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } } }
I20260926 03:43:27.232834 24300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:27.232127 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0285772e3a409994ec271e0d4d4aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0285772e3a409994ec271e0d4d4aff" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39945 } } }
I20260926 03:43:27.233799 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:27.235673 24307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:27.243265 24307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:43:27.255764 24307 catalog_manager.cc:1347] Generated new cluster ID: 5c19800c2086466cbecf59ca4ef31384
I20260926 03:43:27.255941 24307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:27.272534 24307 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:27.273190 24307 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:27.282526 24307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0285772e3a409994ec271e0d4d4aff: Generated new TSK 0
I20260926 03:43:27.283819 24307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:43:27.299803 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24221
W20260926 03:43:27.302296 24283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38693
W20260926 03:43:27.303418 1910 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24221: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:43:27.303664 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24221
2026-09-26T03:43:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2002 ms)
[----------] 18 tests from AuthTokenIssuingTest (37523 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:27 dist-test-slave-lqrk krb5kdc[24319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:27 dist-test-slave-lqrk krb5kdc[24319](info): set up 2 sockets
Sep 26 03:43:27 dist-test-slave-lqrk krb5kdc[24319](info): commencing operation
krb5kdc: starting...
W20260926 03:43:28.364313 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:28 dist-test-slave-lqrk krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:28.668644 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39195
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:29.072892 24332 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:29.073316 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:29.073508 24332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:29.073738 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:29.081738 24332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:29.081878 24332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:29.081971 24332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:29.082050 24332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:29.082149 24332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:29.092913 24332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:29.094348 24332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:29.095822 24332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:29.109661 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:29.110147 24340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:29.110397 24338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:29.111771 24332 server_base.cc:1034] running on GCE node
I20260926 03:43:29.112716 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:29.114492 24332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:29.115788 24332 hybrid_clock.cc:648] HybridClock initialized: now 1790394209115673 us; error 74 us; skew 500 ppm
I20260926 03:43:29.116199 24332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:29.130621 24332 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:29.132628 24332 webserver.cc:460] Webserver started at http://127.1.221.190:37799/ using document root <none> and password file <none>
I20260926 03:43:29.133589 24332 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:29.133731 24332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:29.134120 24332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:29.137456 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "fbde295220974e71a14a546ecbd13180"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "998c476353f6f1a4235fc81c4f138e2b"
server_key_iv: "fc02753e198f8124485dfdd6e3e4523d"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.138281 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "fbde295220974e71a14a546ecbd13180"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "998c476353f6f1a4235fc81c4f138e2b"
server_key_iv: "fc02753e198f8124485dfdd6e3e4523d"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.144603 24332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:43:29.149040 24347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:29.150929 24332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 03:43:29.151139 24332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "fbde295220974e71a14a546ecbd13180"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "998c476353f6f1a4235fc81c4f138e2b"
server_key_iv: "fc02753e198f8124485dfdd6e3e4523d"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.151423 24332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:29.171308 24332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:29.175909 24332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:29.176184 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:29.200124 24332 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39195
I20260926 03:43:29.200235 24398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39195 every 8 connection(s)
I20260926 03:43:29.201817 24332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:29.207419 24399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:29.211967 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24332
I20260926 03:43:29.212258 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:29.213209 1910 external_mini_cluster.cc:1428] Setting key b3a66d4979dcdb8e0975e2366539a401
I20260926 03:43:29.218796 24399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
I20260926 03:43:29.247947 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.224069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.1.221.190:39195)
0926 03:43:29.226036 (+ 1967us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:29.226045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:29.226103 (+ 58us) server_negotiation.cc:407] Connection header received
0926 03:43:29.227879 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:29.227885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:29.228160 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:29.228347 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:29.230064 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.231422 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.232968 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.233586 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.233792 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.238501 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:29.238511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:29.238527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:29.238566 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:29.241585 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.243151 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.243172 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.243176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.243450 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.243990 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.243995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.243999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.244813 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:29.245035 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:29.245913 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 03:43:29.246492 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":1719,"thread_start_us":1459,"threads_started":1}
I20260926 03:43:29.248008 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.224069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39195 (local address 127.0.0.1:59012)
0926 03:43:29.225113 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.225173 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:29.227460 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:29.228391 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:29.228399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:29.229389 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:29.229803 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.229825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.231589 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.231594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.232803 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.232818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.233850 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.233858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.234068 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.234754 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:29.234781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:29.238078 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:29.241834 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.241858 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.241878 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.242901 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.243613 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.243617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.243620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.243857 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.245037 (+ 1180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:29.245045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:29.245373 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 03:43:29.245680 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 03:43:29.246293 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":408,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
I20260926 03:43:29.251977 24399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:29.253343 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.252813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59742)
0926 03:43:29.253051 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.253187 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:43:29.253711 24349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
I20260926 03:43:29.255904 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.255292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59748)
0926 03:43:29.255589 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.255696 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 03:43:29.256124 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:40147: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260926 03:43:29.288450 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:40147
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:43:29.305595 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } attempt: 1
I20260926 03:43:29.307068 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.306485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59760)
0926 03:43:29.306762 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.306916 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:43:29.308295 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.307934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59776)
0926 03:43:29.308091 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.308165 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:43:29.308528 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:40147: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
I20260926 03:43:29.376789 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } attempt: 2
I20260926 03:43:29.377952 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.377488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59792)
0926 03:43:29.377732 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.377816 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:43:29.378863 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.378537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59802)
0926 03:43:29.378673 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.378747 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:43:29.379024 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:40147: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
I20260926 03:43:29.470329 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } attempt: 3
I20260926 03:43:29.471843 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.471259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59816)
0926 03:43:29.471586 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.471700 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 03:43:29.472926 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.472547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59824)
0926 03:43:29.472688 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.472766 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:43:29.473119 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:40147: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
I20260926 03:43:29.616434 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } attempt: 4
I20260926 03:43:29.617730 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.617196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59840)
0926 03:43:29.617490 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.617592 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 03:43:29.619328 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.618877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59846)
0926 03:43:29.619091 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.619191 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 03:43:29.619604 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:40147: Network error: Client connection negotiation failed: client connection to 127.1.221.189:40147: connect: Connection refused (error 111)
W20260926 03:43:29.705425 24406 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:29.705860 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:29.705955 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:29.706056 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:29.714767 24406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:29.714921 24406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:29.714985 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:29.715034 24406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:29.715106 24406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:29.727910 24406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:40147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:29.729789 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:29.731703 24406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:29.746259 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:29.746573 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:29.746821 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:29.748067 24406 server_base.cc:1034] running on GCE node
I20260926 03:43:29.749141 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:29.751062 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:29.752367 24406 hybrid_clock.cc:648] HybridClock initialized: now 1790394209752262 us; error 57 us; skew 500 ppm
I20260926 03:43:29.752904 24406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:29.757165 24406 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260926 03:43:29.759519 24406 webserver.cc:460] Webserver started at http://127.1.221.189:37007/ using document root <none> and password file <none>
I20260926 03:43:29.760473 24406 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:29.760604 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:29.761006 24406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:29.765043 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data/instance:
uuid: "fe91138a07a444f5a3e1532dd59ef7fa"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "5577bc838fbb534a32e49623accf1f00"
server_key_iv: "4b43fe80b7811928418d97944eb41e68"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.766119 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal/instance:
uuid: "fe91138a07a444f5a3e1532dd59ef7fa"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "5577bc838fbb534a32e49623accf1f00"
server_key_iv: "4b43fe80b7811928418d97944eb41e68"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.773245 24406 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 03:43:29.777968 24421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:29.779903 24406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:43:29.780123 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "fe91138a07a444f5a3e1532dd59ef7fa"
format_stamp: "Formatted at 2026-09-26 03:43:29 on dist-test-slave-lqrk"
server_key: "5577bc838fbb534a32e49623accf1f00"
server_key_iv: "4b43fe80b7811928418d97944eb41e68"
server_key_version: "encryptionkey@0"
I20260926 03:43:29.780447 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:29.807279 24406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:29.813464 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:29.813809 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:29.836759 24406 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:40147
I20260926 03:43:29.836786 24472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:40147 every 8 connection(s)
I20260926 03:43:29.838608 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
I20260926 03:43:29.842573 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24406
I20260926 03:43:29.842799 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-1/wal/instance
I20260926 03:43:29.843088 24473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:29.843328 1910 external_mini_cluster.cc:1428] Setting key 7f5d96a9a591796018cebc0986e5352a
I20260926 03:43:29.849993 24473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
I20260926 03:43:29.880180 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.853191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.1.221.190:39195)
0926 03:43:29.853869 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:29.853877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:29.855495 (+ 1618us) server_negotiation.cc:407] Connection header received
0926 03:43:29.858491 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:29.858497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:29.858608 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:29.858753 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:29.861185 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.862129 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.864598 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.865121 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.865276 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.870741 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:29.870750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:29.870754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:29.870830 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:29.874556 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.875801 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.875807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.875809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.875968 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.876655 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.876661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.876665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.877328 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:29.877522 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:29.879610 (+ 2088us) server_negotiation.cc:299] Negotiation successful
0926 03:43:29.879862 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
I20260926 03:43:29.881523 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.854148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39195 (local address 127.0.0.1:59014)
0926 03:43:29.855239 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.855308 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:29.858004 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:29.859091 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:29.859104 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:29.859922 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:29.860834 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.860864 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.862371 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.862377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.864362 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.864383 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.865414 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.865422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.865725 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.866619 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:29.866658 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:29.870255 (+ 3597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:29.874843 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.874856 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.874881 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.875624 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.876098 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.876102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.876105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.876346 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.877570 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:29.877581 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:29.878327 (+ 746us) client_negotiation.cc:769] Sending connection context
0926 03:43:29.878793 (+ 466us) client_negotiation.cc:240] Negotiation successful
0926 03:43:29.879663 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":414,"threads_started":1}
I20260926 03:43:29.883914 24473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:29.900741 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.885202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59860)
0926 03:43:29.885460 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.885484 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:29.885635 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:29.887004 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:29.887009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:29.887593 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:29.887752 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.887763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.889271 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.889277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.890165 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.890179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.891059 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.891065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.891260 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.891914 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:29.891935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:29.894454 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:29.897949 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.897956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.897961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.898352 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.898743 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.898750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.898754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.898933 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.899645 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:29.899650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:29.899813 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 03:43:29.899961 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 03:43:29.900286 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:43:29.900832 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.885483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.1.221.189:40147)
0926 03:43:29.886464 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:29.886472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:29.886532 (+ 60us) server_negotiation.cc:407] Connection header received
0926 03:43:29.886600 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:29.886607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:29.886827 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:29.886992 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:29.887939 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.889121 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.890309 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.890900 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.891031 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.894678 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:29.894693 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:29.894710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:29.894741 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:29.897785 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.898491 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.898498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.898500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.898635 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.899048 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.899053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.899057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.899524 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:29.899753 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:29.900127 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 03:43:29.900415 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":750,"thread_start_us":562,"threads_started":1}
I20260926 03:43:29.904253 24473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:29.905550 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.905067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50766)
0926 03:43:29.905242 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.905381 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:43:29.905820 24422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:29.907414 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.907059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50776)
0926 03:43:29.907199 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.907277 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:43:29.907709 24473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260926 03:43:29.918892 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } attempt: 5
I20260926 03:43:29.920383 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:46401
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:43:29 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:29.936261 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.919853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59868)
0926 03:43:29.920143 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.920172 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:29.920354 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:29.920955 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:29.920961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:29.921484 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:29.921762 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.921775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.922986 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.922992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.924264 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:29.924282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.925171 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.925177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:29.925441 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.926224 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:29.926250 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:29.929001 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:29.933442 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.933453 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.933456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.933987 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.934539 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:29.934543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:29.934545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:29.934704 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:29.935423 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:29.935427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:29.935599 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:43:29.935733 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 03:43:29.935991 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 03:43:29.936522 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.920000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.1.221.189:40147)
0926 03:43:29.920239 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:29.920244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:29.920270 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:43:29.920601 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:29.920609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:29.920751 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:29.920871 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:29.921994 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.922813 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.924483 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:29.924988 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:29.925104 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:29.929341 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:29.929347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:29.929352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:29.929409 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:29.933151 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.934149 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.934155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.934158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.934348 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:29.934826 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:29.934831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:29.934834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:29.935259 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:29.935461 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:29.935863 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 03:43:29.936184 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:43:29.937795 24399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:29.938970 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.938514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50790)
0926 03:43:29.938708 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.938828 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:43:29.939971 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.939665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50792)
0926 03:43:29.939801 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.939871 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:43:29.940325 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:29.957058 24473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 1
I20260926 03:43:29.958297 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.957790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50794)
0926 03:43:29.958003 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.958116 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:43:29.959704 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.959238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50796)
0926 03:43:29.959485 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.959588 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 03:43:29.960036 24473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:29.991588 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 1
I20260926 03:43:29.993134 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.992524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50804)
0926 03:43:29.992820 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.992950 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 03:43:29.994329 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:29.994008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50818)
0926 03:43:29.994144 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:29.994222 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:43:29.994558 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:30.028294 24473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 2
I20260926 03:43:30.029883 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.029232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50824)
0926 03:43:30.029565 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.029703 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 03:43:30.031278 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.030822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50836)
0926 03:43:30.031032 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.031138 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260926 03:43:30.031581 24473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:30.062858 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 2
I20260926 03:43:30.064651 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.063896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50848)
0926 03:43:30.064296 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.064436 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 03:43:30.065908 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.065538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50854)
0926 03:43:30.065698 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.065781 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:43:30.066157 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:30.122866 24473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 3
I20260926 03:43:30.124451 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.123837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50870)
0926 03:43:30.124117 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.124248 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 03:43:30.125900 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.125332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50878)
0926 03:43:30.125627 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.125738 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260926 03:43:30.126235 24473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:30.172461 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 3
I20260926 03:43:30.173997 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.173371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50888)
0926 03:43:30.173637 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.173764 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:43:30.175437 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.174862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50894)
0926 03:43:30.175148 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.175257 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260926 03:43:30.175714 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
I20260926 03:43:30.269555 24473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 4
I20260926 03:43:30.271178 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.270523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50896)
0926 03:43:30.270836 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.270974 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:43:30.272603 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.272115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50902)
0926 03:43:30.272348 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.272449 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 03:43:30.272897 24473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
W20260926 03:43:30.310670 24481 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:30.311034 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:30.311146 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:30.311247 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:30.318898 24481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:30.319027 24481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:30.319092 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:30.319169 24481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:30.319263 24481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:30.329943 24481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39195,127.1.221.189:40147,127.1.221.188:46401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:46401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:30.331360 24481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:30.332798 24481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:30.346666 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:30.346612 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:30.346908 24486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:30.347981 24481 server_base.cc:1034] running on GCE node
I20260926 03:43:30.348925 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:30.350948 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:30.352263 24481 hybrid_clock.cc:648] HybridClock initialized: now 1790394210352169 us; error 76 us; skew 500 ppm
I20260926 03:43:30.352708 24481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:43:30.352969 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 4
I20260926 03:43:30.354419 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.353802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50910)
0926 03:43:30.354090 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.354212 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 03:43:30.355583 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.355170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50920)
0926 03:43:30.355333 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.355452 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:43:30.355896 24399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:46401: Network error: Client connection negotiation failed: client connection to 127.1.221.188:46401: connect: Connection refused (error 111)
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394210, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:30.356603 24481 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260926 03:43:30.358265 24481 webserver.cc:460] Webserver started at http://127.1.221.188:36561/ using document root <none> and password file <none>
I20260926 03:43:30.358999 24481 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:30.359094 24481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:30.359472 24481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:30.362540 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data/instance:
uuid: "08ebcfcb4ef941fcb27ea102343a563f"
format_stamp: "Formatted at 2026-09-26 03:43:30 on dist-test-slave-lqrk"
server_key: "321b9ee0a6ada72c40d4d09a7fff88f8"
server_key_iv: "c58374e91172dc75544860b70bd0ef0b"
server_key_version: "encryptionkey@0"
I20260926 03:43:30.363353 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal/instance:
uuid: "08ebcfcb4ef941fcb27ea102343a563f"
format_stamp: "Formatted at 2026-09-26 03:43:30 on dist-test-slave-lqrk"
server_key: "321b9ee0a6ada72c40d4d09a7fff88f8"
server_key_iv: "c58374e91172dc75544860b70bd0ef0b"
server_key_version: "encryptionkey@0"
I20260926 03:43:30.370229 24481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:43:30.374496 24496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:30.376524 24481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 03:43:30.376745 24481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "08ebcfcb4ef941fcb27ea102343a563f"
format_stamp: "Formatted at 2026-09-26 03:43:30 on dist-test-slave-lqrk"
server_key: "321b9ee0a6ada72c40d4d09a7fff88f8"
server_key_iv: "c58374e91172dc75544860b70bd0ef0b"
server_key_version: "encryptionkey@0"
I20260926 03:43:30.376983 24481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:30.399636 24481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:30.404456 24481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:30.404703 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:30.429711 24481 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:46401
I20260926 03:43:30.429755 24547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:46401 every 8 connection(s)
I20260926 03:43:30.431572 24481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
I20260926 03:43:30.437601 24548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:30.437620 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24481
I20260926 03:43:30.437834 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393848592298-1910-0/minicluster-data/master-2/wal/instance
I20260926 03:43:30.438467 1910 external_mini_cluster.cc:1428] Setting key 1831b4ca8c878d066afefab055d5a2d2
I20260926 03:43:30.450130 24548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394210, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:43:30.465332 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.440896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.1.221.188:46401)
0926 03:43:30.441900 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.441914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.441969 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:43:30.443965 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.443995 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.444484 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.444871 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.447129 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.448589 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.450592 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.451452 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.451618 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.456884 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.456895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.456911 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.456961 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.460537 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.461391 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.461399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.461408 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.461617 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.462051 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.462058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.462061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.462857 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.463105 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.463766 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.464140 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":457,"threads_started":1}
I20260926 03:43:30.470474 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.453934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.1.221.190:39195)
0926 03:43:30.455145 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.455150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.455166 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:43:30.455218 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.455222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.455290 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.455407 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.456673 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.457412 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.458815 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.459202 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.459308 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.463068 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.463076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.463082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.463145 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.466640 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.467670 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.467677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.467681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.467905 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.468564 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.468571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.468574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.469070 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.469240 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.469950 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.470172 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":914,"threads_started":1}
I20260926 03:43:30.470678 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.453733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39195 (local address 127.0.0.1:59028)
0926 03:43:30.454469 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.454528 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:30.454676 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:30.455497 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:30.455505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:30.456274 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:30.456530 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.456539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.457574 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.457579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.458666 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.458679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.459454 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.459461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.459609 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.460302 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:30.460324 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:30.462738 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:30.466838 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.466847 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.466854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.467481 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.468090 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.468096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.468100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.468378 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.469313 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:30.469320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:30.469556 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 03:43:30.469761 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 03:43:30.470196 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":319,"threads_started":1}
I20260926 03:43:30.472754 24548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394210, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:30.487195 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.468786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.1.221.190:39195)
0926 03:43:30.469406 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.469410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.469433 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:43:30.469492 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.469496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.469556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.469650 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.471598 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.472545 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.474180 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.474623 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.474735 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.480240 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.480250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.480256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.480309 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.483713 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.484681 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.484689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.484693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.484868 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.485287 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.485293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.485297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.485817 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.486092 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.486507 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.486780 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":326,"threads_started":1}
I20260926 03:43:30.490523 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.473729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:40147 (local address 127.0.0.1:59876)
0926 03:43:30.473943 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.473970 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:30.474111 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:30.475013 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:30.475019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:30.475514 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:30.475729 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.475742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.477110 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.477116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.478593 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.478609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.479781 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.479790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.480048 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.480789 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:30.480815 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:30.483715 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:30.487630 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.487636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.487639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.488202 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.488712 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.488715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.488718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.488875 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.489448 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:30.489453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:30.489657 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 03:43:30.489880 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 03:43:30.490168 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:43:30.490757 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.473764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.1.221.189:40147)
0926 03:43:30.474671 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.474677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.474697 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:43:30.474758 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.474762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.474829 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.474921 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.475952 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.476928 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.478825 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.479561 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.479710 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.483995 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.484002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.484006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.484051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.487326 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.488359 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.488367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.488370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.488595 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.488992 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.488997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.489005 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.489314 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.489473 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.490065 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.490322 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":686,"threads_started":1}
I20260926 03:43:30.492165 24548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394210, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:30.507850 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.493095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50944)
0926 03:43:30.493370 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.493392 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:30.493568 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:30.493927 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:30.493932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:30.494456 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:30.494662 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.494673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.495687 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.495693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.496835 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.496846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.497756 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.497761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.497958 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.498471 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:30.498490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:30.500725 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:30.504454 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.504459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.504463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.505062 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.505457 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.505462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.505466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.505639 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.506750 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:30.506755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:30.506917 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 03:43:30.507064 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 03:43:30.507364 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 03:43:30.507934 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.493336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.1.221.188:46401)
0926 03:43:30.493494 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.493499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.493515 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:43:30.493702 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.493706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.493785 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.493865 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.494804 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.495561 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.496991 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.497568 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.497725 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.500928 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.500933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.500936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.500965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.504245 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.505209 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.505216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.505220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.505340 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.505795 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.505799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.505801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.506182 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.506341 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.507204 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.507627 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":55}
I20260926 03:43:30.508800 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.489387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.1.221.189:40147)
0926 03:43:30.490028 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.490033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.490057 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:43:30.490104 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.490108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.490177 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.490291 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.491963 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.493003 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.495516 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.496200 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.496354 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.501708 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.501716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.501721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.501768 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.505053 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.506032 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.506037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.506040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.506190 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.506692 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.506696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.506699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.506978 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.507255 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.507893 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.508467 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":270,"threads_started":1}
I20260926 03:43:30.514775 24548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: Bootstrap starting.
I20260926 03:43:30.514997 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:30.519637 24548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:30.520957 24548 log.cc:824] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:30.519049 24374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fbde295220974e71a14a546ecbd13180"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59036
I20260926 03:43:30.524618 24548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: No bootstrap required, opened a new log
I20260926 03:43:30.536944 24548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.537508 24548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ebcfcb4ef941fcb27ea102343a563f, State: Initialized, Role: FOLLOWER
I20260926 03:43:30.538067 24548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.539251 24558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.539836 24558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.540886 24548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:43:30.552577 24569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:30.552755 24569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:30.550463 24481 master_runner.cc:186] Error getting master registration for 127.1.221.190:39195: OK, instance_id { permanent_uuid: "fbde295220974e71a14a546ecbd13180" instance_seqno: 1790394209193589 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:43:30.563119 24481 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:43:30.572160 24473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 5
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:43:30.589423 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.573342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.1.221.188:46401)
0926 03:43:30.573608 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.573615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.573636 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:43:30.573700 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.573704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.573792 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.573890 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.574798 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.575715 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.577043 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.577633 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.577753 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.581682 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.581692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.581697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.581783 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.585727 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.586933 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.586939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.586942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.587110 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.587641 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.587647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.587650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.588091 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.588275 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.588761 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.589019 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:43:30.589479 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.573096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50946)
0926 03:43:30.573317 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.573340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:30.573483 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:30.573936 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:30.573941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:30.574448 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:30.574636 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.574646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.575870 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.575877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.576861 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.576876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.577799 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.577805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.578065 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.578752 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:30.578773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:30.581365 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:30.585957 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.585965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.585970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.586737 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.587258 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.587263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.587268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.587514 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.588227 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:30.588232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:30.588448 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 03:43:30.588619 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 03:43:30.589006 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:43:30.596884 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: Bootstrap starting.
I20260926 03:43:30.600956 24473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:30.602281 24473 log.cc:824] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:30.605880 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: No bootstrap required, opened a new log
I20260926 03:43:30.614475 24473 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.614930 24473 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe91138a07a444f5a3e1532dd59ef7fa, State: Initialized, Role: FOLLOWER
I20260926 03:43:30.615597 24473 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.616799 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.617241 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.618297 24473 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:43:30.632537 24583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:30.632917 24583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:30.633139 24399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } attempt: 5
I20260926 03:43:30.628070 24406 master_runner.cc:186] Error getting master registration for 127.1.221.190:39195: OK, instance_id { permanent_uuid: "fbde295220974e71a14a546ecbd13180" instance_seqno: 1790394209193589 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:43:30.643633 24406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 03:43:30 dist-test-slave-lqrk krb5kdc[24319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394209, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
W20260926 03:43:30.657405 24502 tablet.cc:2369] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:30.661634 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.634086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:46401 (local address 127.0.0.1:50954)
0926 03:43:30.634470 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:30.634505 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:30.634650 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:30.635476 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:30.635481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:30.636038 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:30.636272 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.636285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.638024 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.638031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.639675 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:30.639700 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.641307 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.641323 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:30.641563 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.642563 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:30.642625 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:30.646313 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:30.655603 (+ 9290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.655610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.655614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.656358 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.658406 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:30.658412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:30.658416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:30.658682 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:30.660246 (+ 1564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:30.660252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:30.660480 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 03:43:30.660666 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 03:43:30.661021 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 03:43:30.662750 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.634315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.1.221.188:46401)
0926 03:43:30.634926 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.634933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.634953 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:43:30.635017 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.635022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.635110 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.635214 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.636511 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.637794 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.639910 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.640739 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.640887 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.646956 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.646968 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.646975 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.647101 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.654869 (+ 7768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.656733 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.656743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.656748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.657342 (+ 594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.658975 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.658988 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.658992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.659997 (+ 1005us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.660385 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.661235 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.661927 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260926 03:43:30.670913 24399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180: Bootstrap starting.
I20260926 03:43:30.676314 24399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:30.677999 24399 log.cc:824] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:30.683765 24399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180: No bootstrap required, opened a new log
I20260926 03:43:30.692425 24399 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.692891 24399 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbde295220974e71a14a546ecbd13180, State: Initialized, Role: FOLLOWER
I20260926 03:43:30.693600 24399 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.695504 24586 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.696609 24586 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.697036 24399 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:30.697173 24374 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:43:30.697470 24374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:30.701857 24374 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.705804 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbde295220974e71a14a546ecbd13180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe91138a07a444f5a3e1532dd59ef7fa"
I20260926 03:43:30.706529 24448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:30.710858 24448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde295220974e71a14a546ecbd13180 in term 1.
I20260926 03:43:30.712378 24349 leader_election.cc:302] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fbde295220974e71a14a546ecbd13180, fe91138a07a444f5a3e1532dd59ef7fa; no voters:
I20260926 03:43:30.714773 24523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbde295220974e71a14a546ecbd13180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "08ebcfcb4ef941fcb27ea102343a563f"
I20260926 03:43:30.715502 24523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:30.720109 24523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde295220974e71a14a546ecbd13180 in term 1.
I20260926 03:43:30.721277 24374 leader_election.cc:288] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [CANDIDATE]: Term 1 election: Requested vote from peers fe91138a07a444f5a3e1532dd59ef7fa (127.1.221.189:40147), 08ebcfcb4ef941fcb27ea102343a563f (127.1.221.188:46401)
I20260926 03:43:30.729175 24586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:30.739434 24586 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [term 1 LEADER]: Becoming Leader. State: Replica: fbde295220974e71a14a546ecbd13180, State: Running, Role: LEADER
I20260926 03:43:30.740935 24586 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } }
I20260926 03:43:30.753433 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbde295220974e71a14a546ecbd13180. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.754300 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:30.759538 24601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:30.790493 24601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 03:43:30.817437 24427 tablet.cc:2369] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:30.826826 24523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [term 1 FOLLOWER]: Refusing update from remote peer fbde295220974e71a14a546ecbd13180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:30.828363 24593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:30.838763 24448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [term 1 FOLLOWER]: Refusing update from remote peer fbde295220974e71a14a546ecbd13180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:30.840474 24593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:30.844719 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.774376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.1.221.190:39195)
0926 03:43:30.774910 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.774920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.780105 (+ 5185us) server_negotiation.cc:407] Connection header received
0926 03:43:30.788002 (+ 7897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.788010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.788114 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.788195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.799111 (+ 10916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.800317 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.804282 (+ 3965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.804994 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.805110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.823645 (+ 18535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.823655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.823661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.823722 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.827433 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.833899 (+ 6466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.833906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.833909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.834099 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.840250 (+ 6151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.840257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.840260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.840736 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.841066 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.844011 (+ 2945us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.844331 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260926 03:43:30.848508 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.774606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.1.221.189:40147)
0926 03:43:30.775102 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.775109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.780726 (+ 5617us) server_negotiation.cc:407] Connection header received
0926 03:43:30.790682 (+ 9956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.790689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.790797 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.790876 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.797802 (+ 6926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.798914 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.806234 (+ 7320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.806983 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.807096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.828557 (+ 21461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.828565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.828571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.828624 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.832857 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.841355 (+ 8498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.841363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.841366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.841564 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.842615 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.842621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.842625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.843253 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.843747 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.847785 (+ 4038us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.848109 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260926 03:43:30.852716 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:30.778138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.1.221.188:46401)
0926 03:43:30.778375 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:30.778381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:30.798611 (+ 20230us) server_negotiation.cc:407] Connection header received
0926 03:43:30.798788 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:30.798795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:30.798943 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:30.799064 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:30.801888 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.802997 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.807528 (+ 4531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:30.808318 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:30.808446 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:30.834656 (+ 26210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:30.834663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:30.834668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:30.834717 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:30.839131 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.846786 (+ 7655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.846793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.846797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.846992 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:30.850354 (+ 3362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:30.850360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:30.850364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:30.850845 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:30.851130 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:30.851965 (+ 835us) server_negotiation.cc:299] Negotiation successful
0926 03:43:30.852282 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:43:30.882892 24558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbde295220974e71a14a546ecbd13180. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.883553 24558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.892226 24610 mvcc.cc:205] Tried to move back new op lower bound from 7333454687527362560 to 7333454687220432896. Current Snapshot: MvccSnapshot[applied={T|T < 7333454687527362560}]
I20260926 03:43:30.895843 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.896210 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:30.897236 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbde295220974e71a14a546ecbd13180. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.897663 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.898188 24586 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.898577 24586 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:30.902194 24611 mvcc.cc:205] Tried to move back new op lower bound from 7333454687527362560 to 7333454687220432896. Current Snapshot: MvccSnapshot[applied={T|T < 7333454687527362560}]
I20260926 03:43:30.908623 24601 catalog_manager.cc:1347] Generated new cluster ID: 53a4b3e7a2b1430d83b13b8e3c35f176
I20260926 03:43:30.908785 24601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:30.912860 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.913292 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.917418 24558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbde295220974e71a14a546ecbd13180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbde295220974e71a14a546ecbd13180" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39195 } } peers { permanent_uuid: "fe91138a07a444f5a3e1532dd59ef7fa" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 40147 } } peers { permanent_uuid: "08ebcfcb4ef941fcb27ea102343a563f" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 46401 } } }
I20260926 03:43:30.917788 24558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:30.977983 24601 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:30.980516 24601 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:31.000262 24601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbde295220974e71a14a546ecbd13180: Generated new TSK 0
I20260926 03:43:31.003871 24601 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:43:31.146162 24554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.136749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.1.221.190:39195)
0926 03:43:31.136982 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.136990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.144130 (+ 7140us) server_negotiation.cc:407] Connection header received
0926 03:43:31.144438 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.144445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.144540 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.144611 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.145881 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60236: BlockingRecv error: recv got EOF from 127.0.0.1:60236 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 03:43:31.153682 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.139175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.1.221.189:40147)
0926 03:43:31.139427 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.139433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.151916 (+ 12483us) server_negotiation.cc:407] Connection header received
0926 03:43:31.152187 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.152194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.152292 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.152364 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.153412 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59698: BlockingRecv error: recv got EOF from 127.0.0.1:59698 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 03:43:31.157272 24551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.144947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.1.221.188:46401)
0926 03:43:31.145134 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.145140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.155593 (+ 10453us) server_negotiation.cc:407] Connection header received
0926 03:43:31.155853 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.155859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.155946 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.156014 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.157028 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46096: BlockingRecv error: recv got EOF from 127.0.0.1:46096 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:43:31.181653 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.178757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.1.221.189:40147)
0926 03:43:31.178957 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.178963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.179827 (+ 864us) server_negotiation.cc:407] Connection header received
0926 03:43:31.180177 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.180184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.180322 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.180437 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.181396 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59704: BlockingRecv error: recv got EOF from 127.0.0.1:59704 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:43:31.194049 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.188011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.1.221.189:40147)
0926 03:43:31.188220 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.188226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.192036 (+ 3810us) server_negotiation.cc:407] Connection header received
0926 03:43:31.192349 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.192356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.192453 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.192547 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.193725 (+ 1178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59716: BlockingRecv error: recv got EOF from 127.0.0.1:59716 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:43:31.216235 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.201752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.1.221.189:40147)
0926 03:43:31.201955 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.201962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.213894 (+ 11932us) server_negotiation.cc:407] Connection header received
0926 03:43:31.214647 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.214654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.214760 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.214849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.215979 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59724: BlockingRecv error: recv got EOF from 127.0.0.1:59724 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 03:43:31.233712 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.229680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.1.221.189:40147)
0926 03:43:31.229942 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.229949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.232059 (+ 2110us) server_negotiation.cc:407] Connection header received
0926 03:43:31.232276 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.232282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.232389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.232534 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.233424 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59730: BlockingRecv error: recv got EOF from 127.0.0.1:59730 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260926 03:43:31.248404 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.245984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.1.221.189:40147)
0926 03:43:31.246180 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.246187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.246691 (+ 504us) server_negotiation.cc:407] Connection header received
0926 03:43:31.246922 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.246929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.247082 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.247221 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.248140 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59744: BlockingRecv error: recv got EOF from 127.0.0.1:59744 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 03:43:31.270434 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.268160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.1.221.189:40147)
0926 03:43:31.268372 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.268378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.268877 (+ 499us) server_negotiation.cc:407] Connection header received
0926 03:43:31.269063 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.269070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.269177 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.269296 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.270106 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59748: BlockingRecv error: recv got EOF from 127.0.0.1:59748 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:43:31.311421 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.309220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.1.221.189:40147)
0926 03:43:31.309390 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.309397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.309968 (+ 571us) server_negotiation.cc:407] Connection header received
0926 03:43:31.310128 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.310134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.310239 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.310366 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.311133 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59760: BlockingRecv error: recv got EOF from 127.0.0.1:59760 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 03:43:31.380470 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.378601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.1.221.189:40147)
0926 03:43:31.378808 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.378814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.379066 (+ 252us) server_negotiation.cc:407] Connection header received
0926 03:43:31.379222 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.379228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.379330 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.379474 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.380183 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59772: BlockingRecv error: recv got EOF from 127.0.0.1:59772 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 03:43:31.513779 24556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:31.512050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.1.221.189:40147)
0926 03:43:31.512301 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.512308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.512509 (+ 201us) server_negotiation.cc:407] Connection header received
0926 03:43:31.512650 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.512654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.512729 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.512820 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:31.513549 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59782: BlockingRecv error: recv got EOF from 127.0.0.1:59782 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 03:43:31.554776 24569 catalog_manager.cc:1259] Loaded cluster ID: 53a4b3e7a2b1430d83b13b8e3c35f176
I20260926 03:43:31.554877 24569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: loading cluster ID for follower catalog manager: success
I20260926 03:43:31.557416 24569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: acquiring CA information for follower catalog manager: success
I20260926 03:43:31.559221 24569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08ebcfcb4ef941fcb27ea102343a563f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:43:31.635083 24583 catalog_manager.cc:1259] Loaded cluster ID: 53a4b3e7a2b1430d83b13b8e3c35f176
I20260926 03:43:31.635192 24583 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: loading cluster ID for follower catalog manager: success
I20260926 03:43:31.637784 24583 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: acquiring CA information for follower catalog manager: success
I20260926 03:43:31.639590 24583 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe91138a07a444f5a3e1532dd59ef7fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:43:31.783308 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:31.777304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.1.221.189:40147)
0926 03:43:31.777549 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.777554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.777769 (+ 215us) server_negotiation.cc:407] Connection header received
0926 03:43:31.777901 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.777906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.778061 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.778176 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:31.778696 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.779535 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.781042 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.781498 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.781621 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:31.782175 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:31.782726 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:31.782886 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:31.782986 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:43:31.783068 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 03:43:31.797039 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:31.789627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.1.221.188:46401)
0926 03:43:31.791764 (+ 2137us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.791769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.791788 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:43:31.791924 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.791928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.792092 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.792252 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:31.792708 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.793475 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.794817 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.795216 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.795333 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:31.795879 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:31.796413 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:31.796528 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:31.796632 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 03:43:31.796702 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020,"thread_start_us":1912,"threads_started":1}
I20260926 03:43:31.810698 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:31.803508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.1.221.188:46401)
0926 03:43:31.803901 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.803907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.805386 (+ 1479us) server_negotiation.cc:407] Connection header received
0926 03:43:31.805462 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.805467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.805560 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.805635 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:31.806295 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.807049 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.808772 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.809297 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.809416 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:31.809827 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:31.809974 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:31.810070 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:31.810337 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:43:31.810413 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260926 03:43:31.811403 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:31.803086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.1.221.189:40147)
0926 03:43:31.803308 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.803315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.805514 (+ 2199us) server_negotiation.cc:407] Connection header received
0926 03:43:31.805664 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.805669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.805758 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.805839 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:31.806529 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.807547 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.809560 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.810106 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.810220 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:31.810655 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:31.810786 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:31.810842 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:31.811063 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:43:31.811146 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:43:31.812001 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:31.802990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.1.221.190:39195)
0926 03:43:31.803906 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:31.803913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:31.804516 (+ 603us) server_negotiation.cc:407] Connection header received
0926 03:43:31.804821 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:31.804826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:31.805063 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:31.805243 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:31.805867 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.806894 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.809063 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:31.809632 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:31.809752 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:31.810296 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:31.810778 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:31.810877 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:31.811623 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 03:43:31.811710 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":692,"threads_started":1}
I20260926 03:43:31.817564 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24332
I20260926 03:43:31.818627 24394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:31.928474 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24332
W20260926 03:43:31.947042 24424 connection.cc:535] server connection from 127.0.0.1:59868 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:43:31.947731 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24406
I20260926 03:43:31.948856 24468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:32.058221 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24406
I20260926 03:43:32.075788 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24481
I20260926 03:43:32.077020 24543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:32.196123 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24481
2026-09-26T03:43:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4894 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:43:32 dist-test-slave-lqrk krb5kdc[24696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:43:32 dist-test-slave-lqrk krb5kdc[24696](info): set up 2 sockets
Sep 26 03:43:32 dist-test-slave-lqrk krb5kdc[24696](info): commencing operation
krb5kdc: starting...
W20260926 03:43:33.292168 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/test-user.keytab.
Sep 26 03:43:33 dist-test-slave-lqrk krb5kdc[24696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260926 03:43:33.608906 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43811
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:34.034471 24709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:34.034996 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:34.035095 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:34.035228 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:34.044140 24709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:34.044296 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:34.044364 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:34.044445 24709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:34.044538 24709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:34.056640 24709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:34.058509 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:34.060226 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:34.074503 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:34.074572 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:34.074844 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:34.075776 24709 server_base.cc:1034] running on GCE node
I20260926 03:43:34.076820 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:34.078503 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:34.079818 24709 hybrid_clock.cc:648] HybridClock initialized: now 1790394214079717 us; error 61 us; skew 500 ppm
I20260926 03:43:34.080232 24709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:34.084370 24709 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260926 03:43:34.086411 24709 webserver.cc:460] Webserver started at http://127.1.221.190:35585/ using document root <none> and password file <none>
I20260926 03:43:34.087570 24709 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:34.087713 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:34.088105 24709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:34.092187 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data/instance:
uuid: "85b7ad2141654cb481fe2f136e526e04"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "3abd452952414d46f89a37cda62d2034"
server_key_iv: "7ec9971039efbc5ccebb1236be6fe8c1"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.093223 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance:
uuid: "85b7ad2141654cb481fe2f136e526e04"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "3abd452952414d46f89a37cda62d2034"
server_key_iv: "7ec9971039efbc5ccebb1236be6fe8c1"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.100327 24709 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 03:43:34.105317 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:34.107462 24709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260926 03:43:34.107740 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal
uuid: "85b7ad2141654cb481fe2f136e526e04"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "3abd452952414d46f89a37cda62d2034"
server_key_iv: "7ec9971039efbc5ccebb1236be6fe8c1"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.108040 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:34.129247 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:34.140681 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:34.141032 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:34.161634 24709 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43811
I20260926 03:43:34.161669 24775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43811 every 8 connection(s)
I20260926 03:43:34.163247 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/data/info.pb
I20260926 03:43:34.163746 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24709
I20260926 03:43:34.163980 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-0/wal/instance
I20260926 03:43:34.164628 1910 external_mini_cluster.cc:1428] Setting key 10976f03786b676cd2b01de78c070a1e
I20260926 03:43:34.167733 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:34.174968 24776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
I20260926 03:43:34.203912 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.179213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.1.221.190:43811)
0926 03:43:34.180187 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:34.180202 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:34.180910 (+ 708us) server_negotiation.cc:407] Connection header received
0926 03:43:34.183137 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:34.183143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:34.183474 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:34.183699 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:34.185085 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.186326 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.187911 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.189068 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.189364 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.193912 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:34.193929 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:34.193955 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:34.194072 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:34.197949 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.199237 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.199248 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.199251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.199524 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.200057 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.200061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.200063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.200868 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:34.201047 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:34.201836 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 03:43:34.202450 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":287,"server-negotiator.queue_time_us":743,"thread_start_us":427,"threads_started":1}
I20260926 03:43:34.203986 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.179834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43811 (local address 127.0.0.1:54316)
0926 03:43:34.180721 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.180766 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:34.182709 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:34.183738 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:34.183744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:34.184502 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:34.184853 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.184872 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.186480 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.186485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.187658 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.187675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.189457 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.189463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.189736 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.190541 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:34.190601 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:34.193516 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:34.198220 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.198240 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.198256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.199001 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.199660 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.199666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.199669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.199914 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.201062 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:34.201071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:34.201351 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 03:43:34.201630 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 03:43:34.202269 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":86,"thread_start_us":462,"threads_started":1}
I20260926 03:43:34.207685 24776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:34.209349 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.208633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36828)
0926 03:43:34.208890 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.209081 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 03:43:34.209707 24725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
I20260926 03:43:34.211539 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.210982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36830)
0926 03:43:34.211180 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.211304 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:43:34.211823 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:37843: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260926 03:43:34.235512 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:37843
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:43:34.261353 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } attempt: 1
I20260926 03:43:34.263254 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.262452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36838)
0926 03:43:34.262771 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.262985 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 03:43:34.264863 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.264216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36842)
0926 03:43:34.264458 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.264608 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 03:43:34.265201 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:37843: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
I20260926 03:43:34.333604 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } attempt: 2
I20260926 03:43:34.335464 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.334570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36854)
0926 03:43:34.334986 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.335149 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 03:43:34.336735 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.336224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36860)
0926 03:43:34.336431 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.336541 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 03:43:34.337031 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:37843: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
I20260926 03:43:34.428361 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } attempt: 3
I20260926 03:43:34.430214 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.429372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36866)
0926 03:43:34.429826 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.429973 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260926 03:43:34.431972 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.431238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36874)
0926 03:43:34.431591 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.431720 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
W20260926 03:43:34.432348 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:37843: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
I20260926 03:43:34.575667 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } attempt: 4
I20260926 03:43:34.577450 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.576626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36878)
0926 03:43:34.577074 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.577225 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260926 03:43:34.578824 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.578325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36886)
0926 03:43:34.578517 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.578642 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:43:34.579100 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:37843: Network error: Client connection negotiation failed: client connection to 127.1.221.189:37843: connect: Connection refused (error 111)
W20260926 03:43:34.661453 24783 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:34.661945 24783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:34.662024 24783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:34.662123 24783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:34.670894 24783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:34.671056 24783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:34.671164 24783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:34.671247 24783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:34.671345 24783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:34.688058 24783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:37843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:34.690471 24783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:34.692536 24783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:43:34.707628 24791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:34.707703 24789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:34.707605 24788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:34.709111 24783 server_base.cc:1034] running on GCE node
I20260926 03:43:34.710199 24783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:34.712284 24783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:34.713608 24783 hybrid_clock.cc:648] HybridClock initialized: now 1790394214713508 us; error 55 us; skew 500 ppm
I20260926 03:43:34.714092 24783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:34.718212 24783 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260926 03:43:34.720381 24783 webserver.cc:460] Webserver started at http://127.1.221.189:38773/ using document root <none> and password file <none>
I20260926 03:43:34.721274 24783 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:34.721380 24783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:34.721720 24783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:34.725886 24783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data/instance:
uuid: "f38b4e01d36d4e88be9a708bc07d134a"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "70f7ba73b0ca1354ba7d9bdab08e2a10"
server_key_iv: "cdcc0a004ee5ae80f3f9251c52ed6a08"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.726919 24783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal/instance:
uuid: "f38b4e01d36d4e88be9a708bc07d134a"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "70f7ba73b0ca1354ba7d9bdab08e2a10"
server_key_iv: "cdcc0a004ee5ae80f3f9251c52ed6a08"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.734330 24783 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 03:43:34.739289 24798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:34.741514 24783 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:43:34.741793 24783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal
uuid: "f38b4e01d36d4e88be9a708bc07d134a"
format_stamp: "Formatted at 2026-09-26 03:43:34 on dist-test-slave-lqrk"
server_key: "70f7ba73b0ca1354ba7d9bdab08e2a10"
server_key_iv: "cdcc0a004ee5ae80f3f9251c52ed6a08"
server_key_version: "encryptionkey@0"
I20260926 03:43:34.742106 24783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:34.758262 24783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:34.764245 24783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:34.764595 24783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:34.786315 24783 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:37843
I20260926 03:43:34.786340 24849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:37843 every 8 connection(s)
I20260926 03:43:34.788100 24783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/data/info.pb
I20260926 03:43:34.788730 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24783
I20260926 03:43:34.788926 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-1/wal/instance
I20260926 03:43:34.789552 1910 external_mini_cluster.cc:1428] Setting key 5add90599ae0397e9057b1f09aa4003a
I20260926 03:43:34.793102 24850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:34.801401 24850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:43:34.829097 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.804869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.1.221.190:43811)
0926 03:43:34.805711 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:34.805719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:34.806759 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 03:43:34.809156 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:34.809163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:34.809270 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:34.809381 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:34.811437 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.812276 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.814654 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.815152 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.815289 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.820845 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:34.820853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:34.820857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:34.820930 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:34.824482 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.825736 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.825744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.825748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.825963 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.826536 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.826540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.826544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.826974 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:34.827158 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:34.828353 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0926 03:43:34.828675 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":315,"threads_started":1}
I20260926 03:43:34.829977 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.805541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43811 (local address 127.0.0.1:54332)
0926 03:43:34.806531 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.806585 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:34.808783 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:34.809701 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:34.809711 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:34.810400 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:34.811112 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.811133 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.812479 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.812483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.814449 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.814469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.815486 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.815496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.815874 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.816885 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:34.816924 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:34.820289 (+ 3365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:34.824789 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.824802 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.824825 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.825519 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.826155 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.826159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.826162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.826389 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.827173 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:34.827180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:34.827907 (+ 727us) client_negotiation.cc:769] Sending connection context
0926 03:43:34.828216 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 03:43:34.828751 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":405,"threads_started":1}
I20260926 03:43:34.832242 24850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:34.849222 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.833065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36900)
0926 03:43:34.833258 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.833280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:34.833371 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:34.834539 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:34.834544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:34.835002 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:34.835143 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.835152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.836587 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.836593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.837450 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.837462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.838283 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.838289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.838437 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.839153 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:34.839170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:34.841968 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:34.845932 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.845938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.845941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.846520 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.847011 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.847016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.847018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.847305 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.848117 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:34.848121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:34.848339 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 03:43:34.848504 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 03:43:34.848816 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:43:34.849382 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.833407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.1.221.189:37843)
0926 03:43:34.834120 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:34.834126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:34.834152 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:43:34.834197 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:34.834203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:34.834393 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:34.834526 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:34.835308 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.836458 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.837587 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.838127 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.838235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.842296 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:34.842309 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:34.842325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:34.842362 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:34.845708 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.846690 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.846697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.846700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.846849 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.847472 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.847496 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.847499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.847956 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:34.848145 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:34.848733 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 03:43:34.849016 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":388,"threads_started":1}
I20260926 03:43:34.852854 24850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:34.854387 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.853773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48376)
0926 03:43:34.854007 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.854152 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 03:43:34.854754 24800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260926 03:43:34.856957 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.856406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48382)
0926 03:43:34.856638 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.856772 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:43:34.856962 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
/tmp/dist-test-taskmxwHQo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:37359
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:43:34.857474 24850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:34.878423 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } attempt: 5
Sep 26 03:43:34 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:34.893220 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.879741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.1.221.189:37843)
0926 03:43:34.879986 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:34.879993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:34.880012 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:43:34.880068 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:34.880072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:34.880163 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:34.880251 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:34.881111 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.881824 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.882971 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.883537 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.883659 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.887507 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:34.887513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:34.887516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:34.887551 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:34.890397 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.891166 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.891170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.891173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.891297 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:34.891689 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:34.891692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:34.891694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:34.892004 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:34.892161 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:34.892608 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 03:43:34.892855 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:43:34.893220 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.879495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36902)
0926 03:43:34.879708 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.879727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:34.879849 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:34.880300 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:34.880306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:34.880757 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:34.880961 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.880970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.881970 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.881974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.882823 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:34.882840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:34.883700 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:34.883705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:34.883913 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:34.884741 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:34.884760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:34.887212 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:34.890593 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.890597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.890600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.891030 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.891432 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:34.891436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:34.891438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:34.891580 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:34.892139 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:34.892144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:34.892318 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 03:43:34.892481 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 03:43:34.892836 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:43:34.895028 24776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:34.896795 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.896244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48392)
0926 03:43:34.896464 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.896609 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:43:34.898173 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.897765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48398)
0926 03:43:34.897935 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.898051 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:43:34.898437 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:34.906941 24850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 1
I20260926 03:43:34.908150 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.907693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48408)
0926 03:43:34.907884 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.907985 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:43:34.909590 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.909213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48424)
0926 03:43:34.909405 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.909486 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 03:43:34.909847 24850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:34.949700 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 1
I20260926 03:43:34.951277 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.950644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48426)
0926 03:43:34.950941 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.951086 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 03:43:34.952545 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.952179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48442)
0926 03:43:34.952339 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.952435 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:43:34.952816 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:34.978092 24850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 2
I20260926 03:43:34.979720 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.979056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48444)
0926 03:43:34.979315 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.979536 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 03:43:34.980908 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:34.980480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48454)
0926 03:43:34.980686 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:34.980783 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 03:43:34.981148 24850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:35.021071 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 2
I20260926 03:43:35.022294 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.021805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48462)
0926 03:43:35.022044 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.022130 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:43:35.024000 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.023574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48464)
0926 03:43:35.023769 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.023866 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 03:43:35.024232 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:35.072463 24850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 3
I20260926 03:43:35.074236 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.073718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48480)
0926 03:43:35.073939 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.074044 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 03:43:35.075933 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.075328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48490)
0926 03:43:35.075695 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.075795 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
W20260926 03:43:35.076200 24850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:35.130498 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 3
I20260926 03:43:35.131696 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.131185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48492)
0926 03:43:35.131447 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.131533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:43:35.133280 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.132806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48498)
0926 03:43:35.133070 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.133154 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260926 03:43:35.133569 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
I20260926 03:43:35.219487 24850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 4
I20260926 03:43:35.221072 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.220469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48514)
0926 03:43:35.220757 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.220866 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:43:35.222886 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.222327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48520)
0926 03:43:35.222568 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.222687 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 03:43:35.223201 24850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
W20260926 03:43:35.290503 24858 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:43:35.290868 24858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:43:35.290947 24858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:43:35.291054 24858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:43:35.298905 24858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:43:35.299016 24858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:43:35.299081 24858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:43:35.299127 24858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:43:35.299196 24858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:43:35.310011 24858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43811,127.1.221.189:37843,127.1.221.188:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:43:35.310902 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 4
I20260926 03:43:35.311657 24858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:43:35.312441 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.311869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48528)
0926 03:43:35.312120 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.312233 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:43:35.313349 24858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:43:35.313589 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.313200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48536)
0926 03:43:35.313355 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.313434 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:43:35.313848 24776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:37359: Network error: Client connection negotiation failed: client connection to 127.1.221.188:37359: connect: Connection refused (error 111)
W20260926 03:43:35.327888 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:35.327956 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:43:35.327971 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:43:35.329306 24858 server_base.cc:1034] running on GCE node
I20260926 03:43:35.330404 24858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:43:35.332486 24858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:43:35.333806 24858 hybrid_clock.cc:648] HybridClock initialized: now 1790394215333691 us; error 72 us; skew 500 ppm
I20260926 03:43:35.334375 24858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:43:35.338642 24858 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260926 03:43:35.340852 24858 webserver.cc:460] Webserver started at http://127.1.221.188:44285/ using document root <none> and password file <none>
I20260926 03:43:35.341847 24858 fs_manager.cc:359] Metadata directory not provided
I20260926 03:43:35.341974 24858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:43:35.342386 24858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:43:35.346608 24858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data/instance:
uuid: "c52b2f212d90442482f8f47a4ac19cf2"
format_stamp: "Formatted at 2026-09-26 03:43:35 on dist-test-slave-lqrk"
server_key: "d811109e7f880878991f3fd21267dc1c"
server_key_iv: "bf4e22b4807fb8889ba4789ec0f7d232"
server_key_version: "encryptionkey@0"
I20260926 03:43:35.347842 24858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal/instance:
uuid: "c52b2f212d90442482f8f47a4ac19cf2"
format_stamp: "Formatted at 2026-09-26 03:43:35 on dist-test-slave-lqrk"
server_key: "d811109e7f880878991f3fd21267dc1c"
server_key_iv: "bf4e22b4807fb8889ba4789ec0f7d232"
server_key_version: "encryptionkey@0"
I20260926 03:43:35.355032 24858 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 03:43:35.360105 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:43:35.362212 24858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:43:35.362471 24858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal
uuid: "c52b2f212d90442482f8f47a4ac19cf2"
format_stamp: "Formatted at 2026-09-26 03:43:35 on dist-test-slave-lqrk"
server_key: "d811109e7f880878991f3fd21267dc1c"
server_key_iv: "bf4e22b4807fb8889ba4789ec0f7d232"
server_key_version: "encryptionkey@0"
I20260926 03:43:35.362766 24858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:43:35.392220 24858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:43:35.398145 24858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:43:35.398532 24858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:43:35.421036 24858 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:37359
I20260926 03:43:35.421077 24924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:37359 every 8 connection(s)
I20260926 03:43:35.422780 24858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/data/info.pb
I20260926 03:43:35.427285 24925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:43:35.431372 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu as pid 24858
I20260926 03:43:35.431612 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmxwHQo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393848592298-1910-0/minicluster-data/master-2/wal/instance
I20260926 03:43:35.432166 1910 external_mini_cluster.cc:1428] Setting key f23b3ab455a22252b33515f8384df636
I20260926 03:43:35.435325 24925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:43:35.459540 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.438522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.1.221.190:43811)
0926 03:43:35.439054 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.439061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.440037 (+ 976us) server_negotiation.cc:407] Connection header received
0926 03:43:35.440170 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.440176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.440400 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.440527 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.442423 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.443693 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.446024 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.446640 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.446796 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.451227 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.451234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.451237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.451298 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.454790 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.456162 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.456169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.456172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.456408 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.457020 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.457024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.457026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.457351 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.457514 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.458759 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.459071 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":243,"threads_started":1}
I20260926 03:43:35.461246 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.434516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.1.221.188:37359)
0926 03:43:35.435706 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.435725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.435790 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:43:35.438168 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.438226 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.439015 (+ 789us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.439649 (+ 634us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.441581 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.443696 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.445665 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.446640 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.446898 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.451763 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.451775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.451794 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.451840 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.455951 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.457028 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.457036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.457038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.457168 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.457665 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.457670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.457673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.458274 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.458635 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.459064 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.459425 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":858,"thread_start_us":443,"threads_started":1}
I20260926 03:43:35.461213 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.438477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43811 (local address 127.0.0.1:54336)
0926 03:43:35.439537 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.439606 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:35.439855 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:35.440675 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:35.440685 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:35.441646 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:35.442166 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.442184 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.443924 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.443931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.445825 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.445839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.446954 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.446963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.447275 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.448175 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:35.448218 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:35.450937 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:35.455076 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.455090 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.455107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.455959 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.456592 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.456596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.456599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.456846 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.457490 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:35.457499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:35.458213 (+ 714us) client_negotiation.cc:769] Sending connection context
0926 03:43:35.458598 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 03:43:35.459288 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":73,"thread_start_us":531,"threads_started":1}
I20260926 03:43:35.463457 24925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260926 03:43:35.479820 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.464483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.1.221.189:37843)
0926 03:43:35.465251 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.465255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.465274 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:43:35.465330 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.465335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.465417 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.465559 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.466621 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.467760 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.469215 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.469652 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.469764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.473710 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.473715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.473723 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.473763 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.476485 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.477569 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.477574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.477577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.477721 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.478215 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.478219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.478221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.478544 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.478675 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.479218 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.479460 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":558,"threads_started":1}
I20260926 03:43:35.479960 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.464385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:37843 (local address 127.0.0.1:36910)
0926 03:43:35.464590 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.464612 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:35.464753 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:35.465643 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:35.465649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:35.466190 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:35.466412 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.466423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.468014 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.468019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.469056 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.469074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.469806 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.469811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.470012 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.470586 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:35.470604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:35.473447 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:35.476696 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.476703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.476707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.477384 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.477854 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.477861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.477865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.478064 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.478657 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:35.478662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:35.478893 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 03:43:35.479085 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 03:43:35.479534 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:43:35.481367 24925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:43:35.488636 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.465843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.1.221.190:43811)
0926 03:43:35.466068 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.466075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.466092 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:43:35.466152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.466155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.466230 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.466313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.468003 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.468904 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.470365 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.470842 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.470971 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.481871 (+ 10900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.481880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.481885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.481940 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.485612 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.486470 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.486474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.486477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.486636 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.487019 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.487023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.487025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.487342 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.487721 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.488017 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.488284 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394215, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260926 03:43:35.495535 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.482201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48566)
0926 03:43:35.482455 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.482478 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:35.482632 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:35.483075 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:35.483079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:35.483606 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:35.483805 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.483818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.485269 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.485274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.486236 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.486249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.486952 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.486957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.487130 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.487683 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:35.487699 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:35.490180 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:35.493176 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.493182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.493187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.493532 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.493867 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.493872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.493876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.494005 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.494505 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:35.494509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:35.494662 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 03:43:35.494788 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 03:43:35.495087 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:43:35.495560 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.482340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.1.221.188:37359)
0926 03:43:35.482502 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.482507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.482570 (+ 63us) server_negotiation.cc:407] Connection header received
0926 03:43:35.482790 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.482796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.482891 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.483006 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.483980 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.485104 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.486398 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.486807 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.486927 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.490368 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.490372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.490376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.490403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.493042 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.493636 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.493641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.493645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.493769 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.494099 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.494104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.494112 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.494396 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.494526 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.494877 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.495230 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":64}
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260926 03:43:35.502049 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: Bootstrap starting.
I20260926 03:43:35.505822 24925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:35.507120 24925 log.cc:824] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:35.509074 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.490987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.1.221.189:37843)
0926 03:43:35.491188 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.491195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.491215 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:43:35.491374 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.491407 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.491493 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.491585 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.492952 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.493760 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.495029 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.495571 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.496118 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.501549 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.501555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.501560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.501616 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.505276 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.506346 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.506351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.506354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.506576 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.507107 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.507112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.507115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.507671 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.508037 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.508374 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.508727 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:43:35.510665 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: No bootstrap required, opened a new log
I20260926 03:43:35.515983 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:43:35.518777 24925 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.519280 24925 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52b2f212d90442482f8f47a4ac19cf2, State: Initialized, Role: FOLLOWER
I20260926 03:43:35.520071 24925 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.521700 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.522171 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.522399 24850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 5
I20260926 03:43:35.523131 24925 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:35.518950 24751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "85b7ad2141654cb481fe2f136e526e04"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
W20260926 03:43:35.541038 24944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:35.541263 24944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:35.532950 24858 master_runner.cc:186] Error getting master registration for 127.1.221.190:43811: OK, instance_id { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" instance_seqno: 1790394214155936 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:43:35.545850 24858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:43:35.546262 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.523176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48570)
0926 03:43:35.523689 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.523717 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:35.523916 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:35.524380 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:35.524385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:35.524936 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:35.525309 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.525326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.526939 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.526946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.528430 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.528449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.529903 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.529910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.530197 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.531034 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:35.531064 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:35.535634 (+ 4570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:35.542746 (+ 7112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.542753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.542758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.543432 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.544111 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.544115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.544118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.544317 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.545319 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:35.545325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:35.545507 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 03:43:35.545650 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:43:35.545970 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260926 03:43:35.547087 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.523458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.1.221.188:37359)
0926 03:43:35.523659 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.523664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.523873 (+ 209us) server_negotiation.cc:407] Connection header received
0926 03:43:35.524067 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.524072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.524160 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.524295 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.525539 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.526720 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.528710 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.529676 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.529855 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.536037 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.536046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.536053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.536123 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.542430 (+ 6307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.543659 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.543666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.543670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.543890 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.544522 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.544529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.544532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.545151 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.546497 (+ 1346us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.546615 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.546811 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:43:35.554430 24850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: Bootstrap starting.
I20260926 03:43:35.560514 24850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:35.562466 24850 log.cc:824] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:35.567679 24850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: No bootstrap required, opened a new log
I20260926 03:43:35.581000 24850 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.581612 24850 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38b4e01d36d4e88be9a708bc07d134a, State: Initialized, Role: FOLLOWER
I20260926 03:43:35.582362 24850 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.583926 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.584517 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.585611 24850 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:35.591078 24776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } attempt: 5
Sep 26 03:43:35 dist-test-slave-lqrk krb5kdc[24696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790394214, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
W20260926 03:43:35.611403 24958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:43:35.611677 24958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:43:35.596441 24783 master_runner.cc:186] Error getting master registration for 127.1.221.190:43811: OK, instance_id { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" instance_seqno: 1790394214155936 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:43:35.612558 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.592698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.1.221.188:37359)
0926 03:43:35.592947 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.592954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.592978 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:43:35.593047 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.593053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.593205 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.593321 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.594340 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.595440 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.597438 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.598199 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.598324 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.603354 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.603363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.603370 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.603450 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.607851 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.609217 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.609225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.609230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.609465 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.610075 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.610081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.610085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.610521 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.610693 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.611586 (+ 893us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.611978 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:43:35.612969 24783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:43:35.613519 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.592067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:37359 (local address 127.0.0.1:48572)
0926 03:43:35.592350 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 03:43:35.592378 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:43:35.592555 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:43:35.593369 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:43:35.593375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:43:35.593910 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:43:35.594129 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.594143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.595674 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.595681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.597201 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:43:35.597223 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.598605 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.598612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:43:35.598821 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.599689 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:43:35.599715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:43:35.602970 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:43:35.608135 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.608141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.608145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.608982 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.609656 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:43:35.609662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:43:35.609666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:43:35.609907 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:43:35.610779 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:43:35.610786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:43:35.610993 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 03:43:35.611173 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 03:43:35.613028 (+ 1855us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:43:35.620772 24776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: Bootstrap starting.
I20260926 03:43:35.626034 24776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: Neither blocks nor log segments found. Creating new log.
I20260926 03:43:35.627657 24776 log.cc:824] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: Log is configured to *not* fsync() on all Append() calls
I20260926 03:43:35.631332 24776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: No bootstrap required, opened a new log
I20260926 03:43:35.639122 24776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.639568 24776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b7ad2141654cb481fe2f136e526e04, State: Initialized, Role: FOLLOWER
I20260926 03:43:35.640143 24776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.641398 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.642117 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.642728 24776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:43:35.642794 24751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:43:35.642985 24751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:43:35.645651 24730 tablet.cc:2369] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:35.646908 24751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.650187 24825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85b7ad2141654cb481fe2f136e526e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f38b4e01d36d4e88be9a708bc07d134a"
I20260926 03:43:35.650795 24825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:43:35.651574 24879 tablet.cc:2369] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:35.655054 24825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b7ad2141654cb481fe2f136e526e04 in term 1.
I20260926 03:43:35.656142 24725 leader_election.cc:302] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b7ad2141654cb481fe2f136e526e04, f38b4e01d36d4e88be9a708bc07d134a; no voters:
I20260926 03:43:35.658169 24900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85b7ad2141654cb481fe2f136e526e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c52b2f212d90442482f8f47a4ac19cf2"
I20260926 03:43:35.658782 24900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:43:35.663067 24900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b7ad2141654cb481fe2f136e526e04 in term 1.
I20260926 03:43:35.664443 24751 leader_election.cc:288] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [CANDIDATE]: Term 1 election: Requested vote from peers f38b4e01d36d4e88be9a708bc07d134a (127.1.221.189:37843), c52b2f212d90442482f8f47a4ac19cf2 (127.1.221.188:37359)
I20260926 03:43:35.672084 24961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:43:35.680214 24961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [term 1 LEADER]: Becoming Leader. State: Replica: 85b7ad2141654cb481fe2f136e526e04, State: Running, Role: LEADER
I20260926 03:43:35.681684 24961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } }
I20260926 03:43:35.700686 24965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b7ad2141654cb481fe2f136e526e04. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.701318 24965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:35.720945 24974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:43:35.725775 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.694150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.1.221.190:43811)
0926 03:43:35.694435 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.694442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.702802 (+ 8360us) server_negotiation.cc:407] Connection header received
0926 03:43:35.703329 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.703335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.703469 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.703570 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.705908 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.707060 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.709673 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.710382 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.710484 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.715121 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.715129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.715134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.715187 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.719884 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.722310 (+ 2426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.722317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.722320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.722612 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.723487 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.723508 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.723511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.723950 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.724261 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.725021 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.725300 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:43:35.725764 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.693020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.1.221.189:37843)
0926 03:43:35.693342 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.693363 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.695290 (+ 1927us) server_negotiation.cc:407] Connection header received
0926 03:43:35.696307 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.696315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.696427 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.696511 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.701168 (+ 4657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.702522 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.706442 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.707144 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.707241 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.716397 (+ 9156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.716404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.716409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.716459 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.720796 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.722307 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.722315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.722319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.722872 (+ 553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.723481 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.723487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.723490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.723944 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.724237 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.725013 (+ 776us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.725300 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:43:35.739146 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:35.695438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.1.221.188:37359)
0926 03:43:35.695676 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:35.695682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:35.708468 (+ 12786us) server_negotiation.cc:407] Connection header received
0926 03:43:35.708931 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:35.708939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:35.709100 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:35.709196 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:35.712787 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.714080 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.716439 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:35.717149 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:35.717308 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:35.720625 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:43:35.720634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:43:35.720639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:43:35.720697 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:43:35.726122 (+ 5425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.735833 (+ 9711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.735843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.735847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.736152 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:43:35.736901 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:43:35.736908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:43:35.736912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:43:35.737383 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:43:35.737688 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:35.738352 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 03:43:35.738714 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:43:35.747279 24974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 03:43:35.768165 24804 tablet.cc:2369] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:43:35.782207 24900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [term 1 FOLLOWER]: Refusing update from remote peer 85b7ad2141654cb481fe2f136e526e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:35.784152 24825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [term 1 FOLLOWER]: Refusing update from remote peer 85b7ad2141654cb481fe2f136e526e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:43:35.790771 24965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:35.791826 24965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:43:35.825138 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b7ad2141654cb481fe2f136e526e04. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.825629 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.827591 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b7ad2141654cb481fe2f136e526e04. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.827970 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.831223 24983 mvcc.cc:205] Tried to move back new op lower bound from 7333454707818127360 to 7333454707459969024. Current Snapshot: MvccSnapshot[applied={T|T < 7333454707818127360}]
I20260926 03:43:35.868511 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.869292 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:35.872797 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.873209 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.877885 24989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.878584 24989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04 [sys.catalog]: This master's current role is: LEADER
I20260926 03:43:35.881196 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b7ad2141654cb481fe2f136e526e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b7ad2141654cb481fe2f136e526e04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43811 } } peers { permanent_uuid: "f38b4e01d36d4e88be9a708bc07d134a" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 37843 } } peers { permanent_uuid: "c52b2f212d90442482f8f47a4ac19cf2" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 37359 } } }
I20260926 03:43:35.881538 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:43:35.883145 24974 catalog_manager.cc:1347] Generated new cluster ID: 9ff6125b3d8a4a21aafd82a430e47d1d
I20260926 03:43:35.883291 24974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:43:35.904942 24974 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:43:35.910770 24974 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:43:35.949925 24974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b7ad2141654cb481fe2f136e526e04: Generated new TSK 0
I20260926 03:43:35.951648 24974 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:43:36.125861 24929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.099914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.1.221.190:43811)
0926 03:43:36.100256 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.100264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.123056 (+ 22792us) server_negotiation.cc:407] Connection header received
0926 03:43:36.123824 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.123832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.123953 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.124039 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.125540 (+ 1501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54362: BlockingRecv error: recv got EOF from 127.0.0.1:54362 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260926 03:43:36.129340 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.107413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.1.221.189:37843)
0926 03:43:36.107713 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.107721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.126952 (+ 19231us) server_negotiation.cc:407] Connection header received
0926 03:43:36.127477 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.127484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.127606 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.127693 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.129021 (+ 1328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36924: BlockingRecv error: recv got EOF from 127.0.0.1:36924 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 03:43:36.132670 24928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.130522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.1.221.188:37359)
0926 03:43:36.130748 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.130755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.130855 (+ 100us) server_negotiation.cc:407] Connection header received
0926 03:43:36.131049 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.131056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.131159 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.131331 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.132365 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48588: BlockingRecv error: recv got EOF from 127.0.0.1:48588 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 03:43:36.187846 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.164792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.1.221.189:37843)
0926 03:43:36.165069 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.165076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.184884 (+ 19808us) server_negotiation.cc:407] Connection header received
0926 03:43:36.185658 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.185666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.185842 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.185938 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.187477 (+ 1539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36934: BlockingRecv error: recv got EOF from 127.0.0.1:36934 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 03:43:36.240463 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.227272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.1.221.189:37843)
0926 03:43:36.227523 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.227531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.238231 (+ 10700us) server_negotiation.cc:407] Connection header received
0926 03:43:36.238830 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.238837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.238947 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.239039 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.240200 (+ 1161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36942: BlockingRecv error: recv got EOF from 127.0.0.1:36942 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 03:43:36.285493 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.278012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.1.221.189:37843)
0926 03:43:36.278211 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.278219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.283903 (+ 5684us) server_negotiation.cc:407] Connection header received
0926 03:43:36.284078 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.284083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.284205 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.284338 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.285200 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36956: BlockingRecv error: recv got EOF from 127.0.0.1:36956 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:43:36.294796 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.292851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.1.221.189:37843)
0926 03:43:36.293154 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.293160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.293448 (+ 288us) server_negotiation.cc:407] Connection header received
0926 03:43:36.293634 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.293638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.293712 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.293800 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.294537 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36972: BlockingRecv error: recv got EOF from 127.0.0.1:36972 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260926 03:43:36.309908 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.307076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.1.221.189:37843)
0926 03:43:36.307312 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.307317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.308012 (+ 695us) server_negotiation.cc:407] Connection header received
0926 03:43:36.308230 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.308236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.308383 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.308530 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.309599 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36980: BlockingRecv error: recv got EOF from 127.0.0.1:36980 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:43:36.331867 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.329812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.1.221.189:37843)
0926 03:43:36.330024 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.330030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.330350 (+ 320us) server_negotiation.cc:407] Connection header received
0926 03:43:36.330520 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.330525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.330640 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.330767 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.331618 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36992: BlockingRecv error: recv got EOF from 127.0.0.1:36992 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 03:43:36.369825 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.367877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.1.221.189:37843)
0926 03:43:36.368100 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.368107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.368455 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:43:36.368635 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.368640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.368766 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.368886 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.369580 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37006: BlockingRecv error: recv got EOF from 127.0.0.1:37006 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:43:36.445600 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.443426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.1.221.189:37843)
0926 03:43:36.443627 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.443633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.444164 (+ 531us) server_negotiation.cc:407] Connection header received
0926 03:43:36.444364 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.444369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.444464 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.444566 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.445370 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37008: BlockingRecv error: recv got EOF from 127.0.0.1:37008 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:43:36.543450 24944 catalog_manager.cc:1259] Loaded cluster ID: 9ff6125b3d8a4a21aafd82a430e47d1d
I20260926 03:43:36.543555 24944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: loading cluster ID for follower catalog manager: success
I20260926 03:43:36.546046 24944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: acquiring CA information for follower catalog manager: success
I20260926 03:43:36.548008 24944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c52b2f212d90442482f8f47a4ac19cf2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:43:36.579200 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:43:36.577347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.1.221.189:37843)
0926 03:43:36.577553 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.577557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.577861 (+ 304us) server_negotiation.cc:407] Connection header received
0926 03:43:36.578029 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.578034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.578115 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.578206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:43:36.578936 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37012: BlockingRecv error: recv got EOF from 127.0.0.1:37012 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 03:43:36.613842 24958 catalog_manager.cc:1259] Loaded cluster ID: 9ff6125b3d8a4a21aafd82a430e47d1d
I20260926 03:43:36.613945 24958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: loading cluster ID for follower catalog manager: success
I20260926 03:43:36.616443 24958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: acquiring CA information for follower catalog manager: success
I20260926 03:43:36.618322 24958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f38b4e01d36d4e88be9a708bc07d134a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:43:36.844363 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:36.838966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.1.221.189:37843)
0926 03:43:36.839167 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.839173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.839278 (+ 105us) server_negotiation.cc:407] Connection header received
0926 03:43:36.839503 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.839508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.839668 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.839771 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:36.840176 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.840863 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.842194 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.842592 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.842705 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:36.843187 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:36.843786 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:36.843922 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:36.844015 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 03:43:36.844102 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:43:36.856683 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:36.850501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.1.221.188:37359)
0926 03:43:36.851415 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.851421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.851444 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:43:36.851562 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.851566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.851755 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.851916 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:36.852367 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.853097 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.854524 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.854928 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.855035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:36.855509 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:36.856051 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:36.856154 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:36.856336 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 03:43:36.856412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":651,"threads_started":1}
I20260926 03:43:36.870461 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:36.862758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.1.221.189:37843)
0926 03:43:36.862966 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.862972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.864176 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 03:43:36.864457 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.864463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.864570 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.864657 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:36.865305 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.866280 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.868389 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.869062 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.869179 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:36.869730 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:36.869863 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:36.869940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:36.870116 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 03:43:36.870199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:43:36.870705 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:36.864017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.1.221.188:37359)
0926 03:43:36.864200 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.864206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.864902 (+ 696us) server_negotiation.cc:407] Connection header received
0926 03:43:36.865153 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.865158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.865240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.865335 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:36.865896 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.866812 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.868942 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.869409 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.869517 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:36.869965 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:36.870087 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:36.870161 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:36.870344 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 03:43:36.870444 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:43:36.871672 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:43:36.862769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.1.221.190:43811)
0926 03:43:36.865040 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0926 03:43:36.865045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:43:36.865061 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:43:36.865180 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:43:36.865184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:43:36.865388 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:43:36.865561 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:43:36.866138 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.866947 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.869065 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:43:36.869654 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:43:36.869756 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:43:36.870336 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:43:36.870790 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:43:36.870895 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:43:36.871003 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:43:36.871089 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2166,"thread_start_us":2070,"threads_started":1}
I20260926 03:43:36.876261 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24709
I20260926 03:43:36.877238 24771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:36.989208 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24709
W20260926 03:43:37.008679 24876 connection.cc:535] server connection from 127.0.0.1:48572 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 03:43:37.008949 24801 connection.cc:535] server connection from 127.0.0.1:36902 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:43:37.009300 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24783
I20260926 03:43:37.010440 24845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:37.119954 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24783
I20260926 03:43:37.138062 1910 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu pid 24858
I20260926 03:43:37.139258 24920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:43:37.249528 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmxwHQo/build/asan/bin/kudu with pid 24858
2026-09-26T03:43:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5055 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9949 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368623 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST